Debugging: the hidden productivity killer
Recently, Michael Bolton wrote this awesome tip on twitter:
And it got me thinking, how minor changes in the way we ask questions impact the answers we receive. As a company that offers developer-productivity tools that are often used for debugging, I’m surprised how frequently engineers and their managers underestimate the time and effort, blood and tears they spend on debugging.
Every engineering organization has its status meetings. They might be daily stand up meetings with tech leads and product managers going through Jira. They might be held in more official settings in meeting rooms with powerpoint presentations and managers. Either way, I’m hoping you are not spending too much of your time in meetings.
As far as I can tell, software engineering projects are always late. I have personally missed more delivery deadlines then I could ever count. And in those status meetings, we get the fun task of explaining to our superiors and colleagues just why we are late.
So what do we say? Well, pick your fighter:
- “The feature was developed, it’s just not working as expected.”
- “Getting the environment up and running was trickier than expected.”
- “I had to get CI green.”
- “I had to fix a couple of bugs so work on the feature was delayed.”
All of those answers/excuses share a common trait – writing code is much easier than getting it to do EXACTLY what you want. And that effort, of taking code and making it perfect – well, that’s debugging.
Speaking of CI, how stable is yours? How stable are your unit tests? With modern cloud computing and orchestration, we can hardly blame the infrastructure in how slow and finicky our tests are.
If you have to run your tests multiple times just to get them to pass, then whether you admit it or not, bugs are lurking within your tests and within the very code they are testing. OK then, why doesn’t anybody just go ahead and fix those bugs?
Well, getting tests, especially complex E2E tests to be 100% predictable is quite a challenge. Understanding (i.e. debugging) exactly what’s going on in remote CI environments is no fun at all. Some teams spend weeks getting their tests stable and their CI reliable. Many developers find it easier to just grind their teeth and run the CI yet again.
Waiting for Deployments
How often do your developers wait for deployments? Even if you are an elite DevOps performer, and you are deploying multiple times a day, your developers are still waiting hours on deploy. In most companies, they often wait for days or more. Getting your code deployed to a remote environment such as staging and production and collecting feedback is of paramount importance to a healthy software development lifecycle.
It’s possible that your developer is missing a piece of data to understand a bug, so he’s waiting on that new log line he deployed. Maybe he’s even braver and decided to just go ahead and deploy a potential fix to try to see if this will solve the issue, as data collection is just too hard.
If your developers knew what was happening in remote environments, they wouldn’t have to continuously wait on deployments just to learn more.
Once your application moves from an innovation project to a service that provides business value to your customers, that’s when the real work begins. You get the fun task of supporting them as they grope blindly through your product, failing to grasp its very meaning. And they love nothing more than reporting those pesky little bugs and sending their account executives to hunt you down until they are fixed.
In a high-performance dev team, unplanned work can account for up to 20-30% of total work performed. If your team is underperforming, or if you just rolled out a big new release such as a monthly release or a new product launch, the numbers are likely to be much higher.
In all of the cases mentioned above, your developers are hurting as they spend time, effort and mental energy fighting through the bugs in their codebase. In others, debugging feels so impossible, they barely even try.
If you are a developer, a tech lead or an engineering manager, look out for those signs. Debugging is a core capability in any software engineering organization. You have to make sure your team has the skills and tools to tackle those challenges head-on.