[dropshadowbox align=”none” effect=”lifted-both” width=”auto” height=”” background_color=”#ffffff” border_width=”1″ border_color=”#dddddd” ]tl;dr: We fixed the vulnerabilities. If you’d like to read the conclusion of this admittedly long saga, scroll down to the Conclusion at the end.[/dropshadowbox]

Overview

This month, the Asterisk project performed two security releases to address an unauthorized RTP data disclosure vulnerability in its real-time transport protocol (RTP) stack. If a malicious actor knew the RTP ports for a session, or simultaneously sent packets to all potential RTP ports, and could send enough RTP packets in an established stream, then Asterisk would lock onto the malicious actor as the RTP source. This potentially allowed a malicious entity to temporarily deny, redirect, or capture an RTP stream in an established call, or perform a distributed denial of service (DDOS) attack on the system. Official announcements of the vulnerabilities can be found in AST-2017-005 and AST-2017-008.

While security vulnerabilities can happen in any software project, the fact that this particular vulnerability resulted in an incomplete fix that necessitated a second security release has led us to want to describe in detail what occurred.

A retrospective of the event, if you will.

In this blog post we are going to explore two different topics. First, we’ll look at why the various RTP vulnerabilities happened and what has been done in the Asterisk project to resolve them. Second, we’ll discuss how the project responds to security vulnerabilities and why the project had to release two versions to fix this vulnerability.

As an aside, we will go into some fairly technical details of RTP itself, as well as how Asterisk handles RTP. If you need an overview or a refresher on RTP and the various other VoIP protocols that are tightly related to it (SIP and SDP), you may want to read any or all of the following:

• Wikipedia’s overview of SIP, SDP, and RTP.
• How a SIP call is established.
• A tutorial/walk through of how SIP and RTP are related.

Background

Before we dig into the specifics of the vulnerability, it is important to understand some of the issues that surround RTP as a protocol, as well as the various mechanisms that exist in Asterisk to work with and around those limitations.

As a protocol, RTP presents VoIP developers with some well known challenges that apply to any VoIP system – not just Asterisk. While this list is not exhaustive, the following three combined create the situation which led to the vulnerability in Asterisk. This includes:

• SIP message traffic, including the SDP body which conveys critical information about the RTP stream, is often transmitted in plain text over UDP or TCP.
• Many networks utilize NAT. This causes the offered RTP destination IP address and port conveyed in an SDP Offer/Answer to be unreachable from the other endpoint in a session.
• RTP itself has no built-in authentication or authorization provisions. As such, there is no way to verify from an RTP packet that the packet came from an endpoint authorized to transmit RTP to you on this session.

We’ll explore each of these in detail.

Problem: SIP Message Traffic is (often) Conveyed in Plain Text

In many deployments, SIP phones communicate by sending their SIP messages in plain text over UDP or TCP. If a malicious entity managed to insert themselves into the message stream, they could determine the source/destination addresses for the RTP media stream. While most information in a SIP or SDP packet would be considered metadata,  in a larger context, this information makes it far easier to perform a large variety of VoIP attacks. Without this information, an attacker would have to continually send a large number of packets over a large port range in order to disrupt RTP media streams, which is often infeasible but certainly possible.

Solution: Encrypt the Signaling

Asterisk has, for many, many years, supported encrypting its SIP message traffic via TLS in its chan_sip channel driver. With the introduction of the chan_pjsip channel driver and its accompanying SIP stack, this same functionality was carried over. Thus, if your endpoint supports communicating with Asterisk via TLS, all of your SIP message traffic and the SDP message bodies can be encrypted, and malicious attackers cannot intercept the messages to gain information about the resulting RTP session.

Problem: No Authentication/Authorization of RTP

While SIP messages can be authenticated, the RTP protocol by itself – with no help from other protocols – does not define any authentication provisions. Without involving some other protocols, there is no foolproof way to determine if the RTP packet you received is from a valid sender. The RTP RFC does make some recommendations that mitigate this in some circumstances:

• Keep track of who has sent RTP packets.
• If we start to see RTP packets from a new source, don’t accept them until they’ve sent a certain number of packets.
• If you have a valid source, depending on the application, you may drop any packets that are from a new source that you don’t expect.

Outside of this mechanism, the confidentiality of RTP packets can be provided by using SRTP with one of the key exchange protocols, either SDES or DTLS. In the case of SDES, the key exchange occurs in the SIP message traffic, thus also necessitating the use of encrypted SIP message traffic. In the case of DTLS, the key exchange occurs within the RTP stream. Furthermore, one can use ICE’s ice-ufrag  and ice-pwd  attributes with encrypted signaling to ensure integrity of the RTP stream.

Solution: Encrypt the Media and use strictrtp/probation

Since version 1.8, Asterisk has supported SDES-SRTP, and since version 11, Asterisk has supported both DTLS-SRTP and ICE. With the media stream encrypted, it is extremely difficult if not impossible for a malicious actor to affect the RTP stream.

[dropshadowbox align=”none” effect=”lifted-both” width=”auto” height=”” background_color=”#ffffff” border_width=”1″ border_color=”#dddddd” ]It is important to note that if SDES-SRTP or DTLS-SRTP is combined with encrypting SIP message traffic with TLS, the vulnerabilities reported against Asterisk are exceptionally difficult to exploit.[/dropshadowbox]

Asterisk provides mechanisms that should always be used to help prevent unauthorized RTP traffic from being processed within a session:

• strictrtp – introduced in Asterisk 1.6, strictrtp causes Asterisk to drop any RTP packets that it receives that are not from the source IP address and port of the RTP stream. In effect, once Asterisk has “locked” onto a stream of RTP packets for a particular session, it will disallow packets from any other source (malicious or otherwise).
• probation – introduced in Asterisk 1.8.10.0 and Asterisk 11.0.0, probation further enhances strictrtp by having Asterisk “learn” the source IP address and port of the RTP stream before locking down and rejecting any RTP packets from other sources. This prevents a malicious actor from sending an RTP packet before the first valid RTP packet and effectively “stealing” the stream. The number of packets that Asterisk receives before it locks the source IP address and port, by default, is set to 4.

Problem: NAT

Consider a simple NAT scenario illustrated below, where a SIP phone behind a NAT wants to communicate with an Asterisk PBX with a public IP address:

When the SIP endpoint with its private IP address of 192.168.0.2 sends a SIP INVITE request to the Asterisk PBX, it will denote in the SDP body that it wishes to receive RTP packets on IP address 192.168.0.2 and some port (say 10004), e.g.,

c=IN IP4 192.168.0.2
m=audio 10004 RTP/AVP 0 8 101

Clearly Asterisk can’t send media to 192.168.0.2. It has no routable path to that IP address, and it may even have multiple phones communicating with it that are advertising that same IP address! While there are mechanisms outside of Asterisk that can alleviate issues with NATs (such as SIP ALGs, ICE/STUN/TURN, etc.), these mechanisms may not always be available.

Solution: Symmetric RTP options

While some NAT devices may implement a SIP ALG to assist with advertising the correct IP address and port (and, in some cases, to make things much worse), and while there are some protocols that endpoints may use to help discover their publicly accessible IP address and port, often these things are not available or sufficient for media transmission. In those cases, for a given session, Asterisk provides the ability in both chan_sip (nat=yes or nat=comedia or nat=auto_comedia) and chan_pjsip (rtp_symmetric=true) to send RTP packets to the same IP address and port that we received RTP packets from. Typically, NAT devices will relay packets through that port back to the endpoint that sent the media. Most endpoints, if they receive an RTP packet on the same port that they transmitted RTP from, will recognize that they are behind a NAT and process the media as if it were received on the advertised port. This allows endpoints to punch a hole through the NAT, establishing bidirectional media without the assistance of an ALG or other protocols.

The Vulnerability: Or What Went Wrong

The vulnerability reported to the project disclosed that the probation mode in Asterisk was malfunctioning. For a reason we will get into in a little bit, Asterisk had been changed in Asterisk 11.4.0 and later to have a second probation mode that was always attempting to learn a new source IP address and port for its RTP streams. If a malicious attacker flooded Asterisk with RTP packets (greater than the value of probation within an RTP stream), Asterisk would lock onto the malicious actor as the RTP source. In all scenarios, the non-hijacked leg of the call in Asterisk will hear garbled audio. For the hijacked leg of the call, there are two possible scenarios, depending on Asterisk’s configuration:

• If nat=no (chan_sip) or rtp_symmetric=false (chan_pjsip), the valid actor will continue to hear the original audio stream. This is bad, and potentially very confusing for both valid parties, but the malicious actor doesn’t receive anyone’s RTP stream.
• If nat=yes|comedia|auto_comedia (chan_sip) or rtp_symmetric=true (chan_pjsip), then the valid actor will stop receiving RTP from Asterisk and the malicious actor will receive the RTP stream from the other call leg. This is obviously very bad, as it allows a malicious actor to potentially impersonate the person the valid actor was attempting to call.

Clearly, the probation mode option was broken between its introduction in 11.0.0 and 11.4.0. Why did it break?

As is so often the case with security vulnerabilities, the vulnerability was introduced as a result of a fix to a bug found in testing. The scenario we were attempting to address was, admittedly, rather challenging:

• A Phone (#1) communicating with an Asterisk server (#1), where all media has to flow from Phone #1 through Asterisk #1.
• Asterisk server #1 communicating with a second Asterisk server (#2), where any communication established between the two will result in the Asterisk servers attempting to bridge the media using direct media, that is, the Asterisk servers will attempt to remove themselves from the media path.
• Asterisk server #2 communicating with a second Phone (#2), where any communication established between the two will also result in the Asterisk server attempting to bridge the media using direct media.

When a call is established from Phone #1 through both Asterisk servers to Phone #2, the resulting media flow should look like what is shown in the diagram below:

 

Unfortunately, that didn’t always happen. Under direct media, Asterisk cannot start to remove itself from the media path until it has received the 200 OK SIP response. To start direct media, Asterisk will send re-INVITEs to each endpoint indicating that they should send their media to each other as opposed to Asterisk. While this is happening, however, Asterisk will forward RTP packets between both endpoints, so that there is minimal interruption in media. Using the example above, this looks like the following:

When Asterisk #1 receives a re-INVITE from Asterisk #2, it knows that the source of its media is going to change, and so it tells the RTP stack to “re-learn” the source of the media. However, Asterisk #2 is still sending RTP media to Asterisk #1, and will continue to do so until Phone #2 removes itself from the call flow. So long as Phone #2 starts sending its RTP traffic to Asterisk #1 relatively quickly this isn’t a problem. When Asterisk #2 receives the 200 OK from Phone #2, it will stop sending RTP to Asterisk #1, Phone #2 will be sending its RTP to Asterisk #1, and Asterisk #1 will re-train onto Phone #2 as the media source.

Unfortunately this didn’t always work. Many phones took a substantial amount of time to transmit the 200 OK back to Asterisk #2, causing Asterisk #1 to re-train back onto Asterisk #2 and rejecting the RTP packets later received from Phone #2. When Asterisk #2 drops out of the media path, the end result is one-way audio.

The fix seemed rather simple: rather than stopping our “learning period” in Asterisk, we would continue to remember other sources of RTP even after locking onto a valid RTP source. So long as the current RTP source didn’t stop sending packets, we would not switch to the other sources, but if our source of media went away, we could – within the number of packets specified by probation – switch to another sender.

That works just fine when all of the RTP packets are arriving at approximately the same rate. What we didn’t consider was that a malicious actor may send probation + 1 packets in between the time a valid RTP source sends two packets. Because Asterisk is always learning, this would cause Asterisk to switch the RTP source to the malicious actor.

Hence, the vulnerability.

The Security Fixes

Fix #1: AST-2017-005

The patch that fixed the vulnerability did so by addressing the flooding shortcoming in the original bug fix. While the RTP stack is learning about its source of media, if a source sends multiple RTP packets too fast, they are discarded. This “flood protection” keeps a malicious attacker from injecting  packets between two valid RTP packets and setting itself as the valid media source.

The first fix for the vulnerability was released in versions 11.6-cert17, 11.25.2, 13.13-cert5, 13.17.1, and 14.6.1.

Fix #2: AST-2017-008

Unfortunately, the first fix for the security vulnerability missed two issues:

• While RTP packets would no longer re-train to a malicious attacker’s source address, RTCP packets could still be stolen using the same packet flooding approach.
• The protection afforded by the strictrtp setting only started after the first RTP packet that was received. If a malicious actor could send an RTP packet before any RTP was transmitted by the valid source, it could still set itself as the valid media source.

While still very serious, these issues are somewhat less severe in nature:

• While stealing RTCP packets is certainly not good, a malicious actor cannot perform any adverse actions within Asterisk or otherwise affect the call with the RTCP packets. The RTCP packets sent by Asterisk only contain call quality metrics, and Asterisk only uses RTCP packets for reporting purposes. Hence, while this is a valid vulnerability, there is very little practical impact from its exploitation.
• The issue where Asterisk would lock onto the first RTP packet received as a valid source is much more serious. However, this can be a challenging vulnerability to exploit:
  • If an attacker has managed to insert themselves into the SIP traffic flow (a “man in the middle” attack), the attacker must send an RTP packet faster than the valid source. While feasible, it is harder to exploit than the initial vulnerability, where the attacker could initiate a packet flood at any point in time during the call and steal the RTP stream.
  • If an attacker has not managed to insert themselves into the SIP traffic flow, they would have to constantly bombard Asterisk’s RTP port range in the hope that they can hit a valid call and be the first RTP packet to arrive. For Asterisk’s default port range, and for commonly used codecs, this would require constantly sending anywhere between 28 MB/sec and 168 MB/sec of RTP packets.

While less severe, these were still flaws in the original fix. The patch released under AST-2017-008 addressed both parts:

• RTCP packets are now provided the same flood protection as RTP packets.
• The first RTP packet is now subjected to the probation learning period and the rest of the protection afforded by strictrtp, including the new flood protection.

The second fix for the vulnerability was released in versions 11.6-cert18, 11.25.3, 13.13-cert6, 13.17.2, and 14.6.2.

Disclosure Problems

Given all of that, why were the issues with the first patch not found during testing?

To answer that, it’s instructive to walk through how security vulnerabilities are reported to the project, and the specific events that occurred with this vulnerability report.

The Asterisk project has a long history of treating security vulnerabilities seriously and responding responsibly to security issues within the project. We have:

• A documented process on the Asterisk wiki outlining how to report a vulnerability responsibly to the project.
• An e-mail address (security@asterisk.org) that anyone can use to contact the project maintainers when they believe they have found a security issue.
• Public disclosures of all security vulnerabilities on asterisk.org.
• A public mailing list that users can subscribe to for security notifications.
• An archive of all security vulnerabilities and patches for the supported versions of Asterisk when the vulnerabilities were fixed.
• Public disclosure (with CVEs) of security vulnerabilities in Asterisk on other public mailing lists, including full-disclosure

Behind the scenes, we also try to work with open source projects that depend on Asterisk and commercial companies who have built products on top of Asterisk. The upside of this is that the entire ecosystem is ideally ready when an Asterisk security release is made; the downside is that sometimes this accommodation can lengthen the time it takes for the Asterisk developers to publicly release the fix for the vulnerability.

When a patch is being developed to fix a security vulnerability, the Asterisk team tests the patch against the automated tests in the Asterisk Test Suite, as well as against the scenario outlined by the vulnerability reporter. Once we feel that we have resolved the issue, the team provides the patch to the security reporter for independent verification that the vulnerability is fixed. This trust is essential to the process. While we’d all like to believe that we’re infallible, the fact that a vulnerability occurred means that we are not. Often, vulnerability reporters will have extensive fuzzing suites and other tools used to produce the vulnerability that are not available to the Asterisk team. It’s not uncommon for security reporters to have other attack vectors that an initial patch misses simply because the Asterisk team does not have the same reproduction conditions as the reporter. Fixing a vulnerability is often a collaborative process, and we’ve had great success working with security researchers. Having the vulnerability reporter sign off on the fix helps us ensure that when we go public with the fix, the vulnerability is well and truly resolved.

Once a patch is verified by all parties, the Asterisk team will coordinate with the vulnerability reporter when the release will be made. Sometimes this is relatively quick. Sometimes, due to the coordination with other projects and companies, it can be much longer. This is traditionally a collaborative process, and if a vulnerability reporter wants to disclose the fix quickly, the Asterisk team will attempt to meet their requirements.

In this particular case, things didn’t go quite as smoothly:

1. When the patch was ready, the Asterisk team requested three months to make the security release. This length of time was requested because a company that builds a product on top of Asterisk was caught at a particularly bad time, and the team wanted to give them some relief so that they could adopt the security release appropriately. While this was unfortunate for everyone, when this was communicated to the vulnerability reporters, they accepted that timeline without complaint.
2. The vulnerability reporters informed the Asterisk team that the first fix was sufficient. It was not until after the first security release was made available that the reporters disclosed in their vulnerability report the existence of the flaws in the first fix.
3. Once the Asterisk team realized there were flaws in the first fix, we released the fixes under the second security notice.

As an open source project, we can’t control how people report security vulnerabilities. At any point in time, someone who discovers a security vulnerability in Asterisk can announce it to the world. All we can do is respond as best we can. Hopefully, the turnaround on the second security release confirms the project’s philosophy to take security issues seriously and responsibly.

Conclusion

If you are a user of Asterisk, please rest assured that the Asterisk team will continue to act responsibly with security issues. We encourage everyone – users and vulnerability researchers – to report security issues to the project. We will continue to work with issue reporters to ensure that security fixes are made correctly and as quickly as we can.

And as always, thanks for contributing to the project, in whatever way you choose to do so!