Overview

One of the most challenging things about coding is when your code doesn’t work and you have to figure out why. Trying to track down a problem can be half the battle. You might not know what’s causing the problem, how to fix it, or where to look in the first place. This can be especially frustrating when you’re trying to meet a deadline. The nice thing about debugging though is that there’s always something new to learn, and hopefully this post will teach you something you didn’t know before that will aid you in your future debugging endeavors.

Identify the Problem

Step 1

This is step one: figuring out what the problem is. Debugging should always start here. You could be on a call with someone and then lose audio from one side. Maybe you’re trying to find the speed you need to get an aircraft off the ground, but the simulation never makes it there. Or maybe you’re simply trying to print out “Hello, world!” in your first program and it just won’t compile. What do you do?

Step 2

Replicate the problem. This may not be possible 100% of the time (especially if it’s not you that is experiencing the problem). Of course, you want to have access to the system that is experiencing the problem, but that may not be possible either. The best you can do is set up your environment as close as possible to match the one with the problem. Otherwise trying to solve the issue is more based on speculation and educated guesses rather than getting actual results.

Step 3

Try to isolate the problem. Adding debug statements throughout the code might seem “hacky,” but believe me when I say this is one of the best ways to get an understanding of why something is doing what it’s doing. You might think that path B will never be taken because path A is the only one that makes sense logically, but your code might very well be doing things you wouldn’t expect it to. It may be an off-nominal path, but it’s still a path that can be taken nonetheless.

Final Steps

Never assume anything. Sometimes the problem will be in the place you least expect it to be. While it may not be the first place you look, do look at it at some point. It’s worth testing all angles that may be involved in the scenario. Just because a function returns the correct result 99% of the time, it doesn’t mean that it’s doing the right thing in all cases. Investigate everything worth investigating, and then investigate some more!

Tips and Tricks

Debug Statements

I mentioned this above, but adding debug statements is one of the best things you can do to track down a problem. You can follow these statements through the execution of your code and see if something is happening in the wrong order, not doing what it’s supposed to, or just not happening at all. The other day I was writing a test for some code I had written. Test one passed. Test two passed. Then I got to test three, which was my nominal test case, and I could not get it to pass. The test logic seemed fine, and I was worried that there was something wrong with my code.

I started to add debug statements and quickly found out that something I was expecting to happen was not happening at all. A closer look at my test showed me that, sure enough, the packets I was expecting were never sent in the first place and I needed to add a mechanism to my test that did some extra work in the RTP stack. A few minutes later and viola, my last test now passed along with the other two.

SIPp and Wireshark

Now let’s hop over to issue reproduction. This can be one of the most frustrating things you have to deal with when debugging. You can try to replicate the problem for days and never see the problem. Luckily, there are some tricks you can do to help with issue reproduction. For example, suppose you have some weird audio issues with a call. The audio is coming across patchy, or sounds like the call is breaking up. You might set up your environment to be exactly like the one having this problem, but have no issues whatsoever with your calls.

There’s a cool thing you can do with Wireshark and SIPp, however, that has helped me in the past. If you have the packet capture from the original issue, you can go into Wireshark, open up the pcap file, and extract the audio from both ends of the call and put them into separate files. Then, in SIPp, you can play those audio files to each other through Asterisk and essentially have the exact scenario (same packets and all), and it’s as easy as this:

<nop>
    <action>
        <exec play_pcap_audio="/path/to/pcap"/>
    </action>
</nop>

Pretty cool, right? You can do this in both SIPp scenarios to have them “talk” to each other as if it were any normal call. This is a more advanced debugging technique, but worth taking the time to do with more difficult problems.

The Basics

These may seem more obvious, but I feel obligated to put them here. If it’s a crash, you definitely want a backtrace. Usually this will point you to the exact cause of the problem, and most of the time it’s due to a NULL variable that’s being accessed without checking it first. This is a great way to bypass the debugging phase completely (assuming it points to the problem).

Another thing to be aware of is configuration. There have been several issues I’ve looked at where the problem stemmed from a configuration error rather than a code error. This is an important area to look at when trying to figure out why something doesn’t work, too. It may be that one little thing needs to be changed to make everything work as it should.

Grab a Friend

The last thing I’ll mention is get someone else to look at the code. It’s very easy to get so heavily invested in a problem that you may miss the obvious or feel like you’ve exhausted your ideas and resources. More often than not, someone else can provide some insight to the problem that you may not have been aware of before. This is one of the major benefits of working on a team, and I highly recommend always having someone else to either work with or to review your code.