The Flytrex Debugging Challenge
Debugging in production, without the right tools, can be situations made out of the most well-crafted nightmares. That’s exactly what Flytrex found out when they came across a bug within one of their drones – while it was in flight.
Flytrex faces a few core challenges when it comes to debugging their aircrafts. To begin with, their technology is extremely complex. Second, they
need to abide by stringent FAA regulations. And third, when bugs happen in production, it means that they’re dealing with a bug that can be found in
an airborne system.
Their drones are complex entities that include software, hardware, embedded code, and operations. When previously attempting to debug them, Flytrex had found themselves trying a myriad of methods to debug their drones. This included reproducing their bugs in local or staging environments, reproducing changes that had logging, or redeploying changes to production withthe aim to collect more information about the bug that needed to be fixed. All of these methods took a lot of time and, in most cases, proved to be nearly impossible. This was because a good amount of Flytrex’s issues can only be resolved in the field, as that is where the set of relevant parameters happen. Therefore, reproducing these bugs locally wasn’t the right solution.