Serverless Production Debugging: AWS Lambda Debugging in Production and Elsewhere

Ever feel like the world is moving by so rapidly, that it feels like something got left behind in the rush? That’s how we feel about being able to debug your applications easily. Technology has advanced rapidly, but the ease of debugging has not.

With the introduction of serverless computing, the way enterprises are designed and how they build their production applications was changed. The introduction of Serverless allowed developers to build and ship much faster, which in turn allows them to concentrate on coding rather than maintenance, auto scaling, and server provisioning, to name a few.

There are a multitude of serverless computing options, but for the sake of this article we will focus on AWS serverless applications. In short, an AWS serverless application is made up of multiple components, and the AWS Lambda function is what essentially connects these components and holds them together.

Debugging Differences: Monolith vs. Serverless Applications

The hype around serverless applications and microservices grows every day, and that growth engenders the negativity around the term “monolith”. Yet, as most developers know, when it truly comes down to the nitty gritty of debugging, monoliths win the debugging game with the ease of debugging that they allow. Due to the complexity of its architecture, Serverless Applications are much more difficult to debug. This difficulty has led to many challenges when using Serverless Applications.

Necessities of Debugging a Monolith App

Many developers would acknowledge that debugging is not their favorite part of the job, and when combining debugging with monolith applications, it gives them a moment of pause. There is no need for this though, as monoliths are simpler to debug.

So how does one begin? First start by choosing the preferred IDE, where a breakpoint will be set at the line of code you want to inspect. After the breakpoint is set, the app will then be run in debug mode. In monoliths, most issues must be debugged locally as there are only two failure points: database behavior or UI/backend. Both of these can be set up in a local environment. Debugging monoliths is a bit simpler because all tasks run on a single process.

So What’s Different? The Path to Debugging a Serverless App

As serverless has paved the way for new technology, it has also changed the process of debugging. When debugging a serverless application, the most common method is logging. This is because it allows the developers to understand real application behavior.

Serverless flows have more components, and are therefore more complex. Due to this complexity, they have more failure points. It isn’t possible to set up all of the components in a local environment in order to debug and there is no option for debugging in a remote environment due to the nature of a serverless application.

So what can go wrong? Plenty of things, as it turns out. Serverless applications are complex and have many components, which introduces many opportunities and variables for things to go awry. On top of that, the loose dependency between all these components makes it harder to understand when interface contracts or compatibility is broken, making the developer’s life even more difficult while debugging.

Serverless Debug Mode: The Challenges

A serverless application is complex. It is made up of multiple components in which a developer might have to debug any or all of these components when an issue arises. This can lead to many challenges that are common to any developer that attempts this.

Here are a few examples of these common challenges:

Challenges of Modern Distributed Architecture

The components of serverless applications are highly distributed. Having the ability to locally simulate these components is not always an option. Even more so, debugging them in a live environment is not easy and it can be quite costly. Having distributed architectures leaves room for things to go wrong in many areas of the stack. Having the ability to drill into the code as various services are running in their native environments is critical to quick defect resolution.

Gaps in Remote Serverless Debugging

Remote debugging is not a usual option for serverless applications, as there is a lack of access to server and OS level. When developers aren’t able to understand where the issue is and why it is happening this proves to be a major stumbling block, because they are then unable to debug live, which is necessary in order to fix the issue.

Serverless Functions are Ephemeral

When focusing on serverless computing, servers are ephemeral in their design. This allows for a limited opportunity to identify what the crux of the problem is in order to debug.
As with microservices in general, serverless instances can blink in and out of existence. Different instances of the same function can be invoked for different sequential requests (load-balancing). This already makes it difficult to try and pinpoint which instances should be observed, but ephemerality in Serverless takes a step further by creating the logic that manages these motions that are almost completely invisible to us.

The Price is High

Most debugging is done in a live environment in a serverless application. This can prove to be an expensive process. This is due to the fact that when debugging, log statements need to be added. Once added, they need to be stored somewhere or to be aggregated to a 3rd party resource, such as logz.io or Datadog. Then, of course, comes the wait for redeploys and tests until those are actually operational. Essentially, this whole process and the overhead it causes raises the price of debugging.

Local Setup Frameworks are Scarce

Very few frameworks exist to set up Lambda in local environments. Those frameworks that do exist come with limitations that impact a developer’s ability to debug.

Debugging Serverless Locally with Serverless Framework

The Serverless Framework is an open-source framework that is universally used for developing and then deploying serverless applications. It can be used with other serverless providers because it isn’t closely connected to AWS. The Serverless Framework provides an open-source command line interface (CLI). The CLI  gives support for commands, which allows serverless applications to be created and built using a .yml file template.

Debug Serverless In the Cloud: How To

In this article, so far we have focused on locally debugging a serverless application. The problem is, however, that when serverless applications run in live environments, new issues or bugs can be discovered when the functions of the applications connect to other services and some of the functions aren’t able to be tested locally.

API Testing Tools to Debug Serverless

In order to debug a serverless application on AWS, there is the option to begin with API Gateway testing. There are a variety of clients available in order to test the endpoint, the most common being SoapUI, Postman, or Insomnia.

AWS Console to Debug Lambda Function

A helpful option for testing Lambda functions directly without the involvement of an event source is the AWS Console. This option is useful when having to check real-time issues when deploying a Lambda for the first time.

To begin with, create test events. These events can be run directly on the function while building and writing the application. The test events are able to mimic what is anticipated from any AWS service.

Further metrics are able to be checked by using the monitoring tab. This tab has the ability to provide metrics that check throttling, error rate, etc. This can be very useful when debugging issues. Even more so, detail logs can be viewed by clicking on the “view logs in CloudWatch” button.

Use AWS Cloudwatch Logs to Debug Lambda Function

The primary source of information as to how an AWS application is behaving is AWS CloudWatch Logs. This applies to Lambda functions as well. By default, Lambda Functions send data to CloudWatch, which then creates a LogGroup for each Function. Within each LogGroup, there will be a variety of Log Streams that assemble the logs that are produced from each particular Lambda instance (multiple instances can be created for each Function). Log Streams consist of log events. If you click on an event, it will give you detailed information for the particular Lambda function that was invoked.

This shows the best way to see what’s happening with a specific invocation that has an issue with a Lambda function, when digging through logs to do just that. CloudWatch Logs are almost real time, and therefore as many logs as needed to debug an issue can be collected.

AWS X-Ray to Debug Lambda Function

CloudWatch may have many great features, yet it is limited in providing information for the Lambda Function. So how is that information obtained? X-Ray was introduced by AWS to provide answers to questions, such as what failures are the ones causing performance issues and how to get around integration. It generates a map of application components and also allows end-to-end tracing requests. X-Ray can also be used to analyze apps while they are in development or production.

Leading Tools for Debugging Live Serverless Code in the Cloud

Debugging an AWS application is possible when using native tools, such as X-Ray, SAM, and CloudWatch. Be that as it may, it is often necessary to use multiple tools to get all the information that’s needed. Commercial tools such as Rookout, Lumigo, and Epsagon do away with that need. Instead of getting buried under all these tools, the commercial tools are simpler to use, to set up, and provide a much more seamless experience for tracing, monitoring, and debugging Lambda functions as well as AWS and third-party managed services.

Lumigo

Lumigo is a SaaS platform that helps developers monitor and troubleshoot their serverless & microservices applications. The platform lets developers identify and fix issues in their applications before they impact the end user.

Lumigo was founded by former Check Point executives and is an AWS Advanced Technology Partner.

Epsagon

Epsagon, founded in 2017 by veterans of the Israel Defense Forces’ cyber intelligence unit, has built an AI-powered automated end-to-end performance monitoring platform for serverless architectures that can predict performance issues before they occur, allowing any company – from SMBs to large enterprises – to eliminate downtime by proactively identifying and flagging potential problems.

Rookout

Rookout is a data extraction and pipelining platform, which provides an unparalleled capability to collect any piece of data, from the deepest levels of live code, on demand. Save hours of work and reduce debugging and logging time by 80% – with zero friction, overhead, or risk. Using non-breaking breakpoints, Rookout empowers engineers to find the information they need and deliver it anywhere, in order to understand and advance their software.

Rookout is SoC2 and HIPAA compliant, and is currently available in Python, Node.js, and JVM runtimes, in all environments (from on-prem to Serverless).

The Holy Grails of Serverless AWS Lambda Debugging

Where are we ultimately running to in our haste to debug our applications? Serverless has already proven that it’s paving the way towards revolutionizing the way cloud applications are built and deployed. It solves many problems that developers have, but also introduces many of its own new challenges. Most specifically, these challenges arise in the areas of debugging, tracing, testing, and monitoring. The focus by commercial vendors on the building of tools is done as a result of the need to address these problems. So while you run straight to finally get rid of that pesky bug that’s been (no pun intended) bugging you for weeks: take a step back, relax, sip that cup of coffee, and realize that the tools needed to win the technological race exist and are right at hand.