A Brief History of the Internet’s Biggest BGP Incidents

01/09/2023

Doug Madory -Director of Internet Analysis at Kentik

Originally published in Kentik Blog

Summary

Stretching back to the AS7007 leak of 1997, this comprehensive blog post covers the most notable and significant BGP incidents in the history of the internet, from traffic-disrupting BGP leaks to crypto-stealing BGP hijacks.

---

In the summer of 2022, I joined a team of BGP experts organized by the Broadband Internet Technical Advisory Group (BITAG) to draft a comprehensive report covering the security of the internet’s routing infrastructure. The section that I was primarily responsible for covered the history of notable BGP incidents, a topic I have written about extensively throughout my career in the internet industry.

Below is an edited version of my take on the internet’s most notable BGP incidents. Henry Birge-Lee of Princeton was the primary author of a large portion of the section on the attacks on cryptocurrency services.

BGP routing security incidents in the wild

BGP routing incidents can be problematic for a range of reasons. In some cases, they simply disrupt the flow of legitimate internet traffic while in others, they can result in the misdirection of communications, posing a security risk from interception or manipulation. Routing incidents occur with some regularity and can vary greatly in operational impact. In this blog post, I will address selected specific incidents which have demonstrated the range and gravity of threats to the stability and security of the internet’s routing system.

Disruptions and attacks caused by BGP incidents

In BGP parlance, the term “routing leak” broadly refers to a routing incident in which one or more BGP advertisements are propagated between ASes (Autonomous Systems) in a way they were not intended to. Often these incidents occur accidentally, but malicious actors may also attempt to camouflage intentional attacks under the guise of apparent accidents.

In 2016, RFC 7908 introduced a more complex taxonomy of BGP routing leaks, but in this post, I will employ simply two main categories of error: origination and AS path.

• A mis-origination occurs when an AS originates (announces with its ASN as the origin) a new advertisement of a route to an IP address block over which it does not possess legitimate control, consequently soliciting traffic destined to those IP addresses.
• An AS path error occurs when an AS inserts itself as an illegitimate intermediary into the forwarding path of traffic bound for a different destination.

This distinction is important because the two types of error require different mitigation strategies.

Additional reading:

Analysis of 7 BGP Variables in the Region during 2022

What is the difference between a BGP hijack and a BGP route leak?

Generally the phrase “BGP hijack” often connotes malicious intent, whereas a “BGP route leak” is assumed to be accidental. Complicating matters, there are BGP incidents which involve both intentional and accidental components and some that we simply don’t know whether they were intentional. Experts in this area can hold varying opinions about what constitutes a BGP leak versus a BGP hijack.

BGP origination errors

The AS7007 incident in April 1997 was arguably the first major internet disruption caused by a routing leak. In this incident, a software bug caused a router to announce a large part of the IP address ranges present in the global routing table as if they were originated by AS7007. This origination leak was compounded by the fact that the routes were more-specifics (i.e., smaller IP address ranges) and, therefore, higher priority according to the BGP selection algorithm.

An additional factor contributing to the degree of disruption was the fact that the leaked routes persisted even after the problematic router was disconnected from the internet. During the leak, a large portion of the internet’s traffic was redirected to AS7007, where it overwhelmed its networking equipment and was dropped.

The AS7007 incident was followed soon after by a massive leak from AS701, which was UUNet at the time. In this incident, AS701 originated all of the IPv4 space contained in 128.0.0.0/9 as /24’s disrupting the flow of traffic to a large portion of the global routing table.

In subsequent years, other similarly large origination leaks have occurred, disrupting internet communications. These incidents include the Turk Telecom leak of December 2004, the China Telecom leak of April 2010, and Telecom Malaysia leak of June 2015. Each of these disruptions lasted less than an hour and appeared indiscriminate in the address blocks affected.

Large-scale origination leaks like these have become less frequent in recent years due to increases in the automation of router configuration in topologically-central networks. Two competing methodologies, RPSL and RPKI, are used to inform the defensive configuration of routers. In both cases, information pairing IP address blocks with authorized origin ASNs (Autonomous System Numbers) is made public, and is distilled by most large network’s operators into “filter lists,” which block the assimilation of nonconforming BGP route advertisements into local routing tables.

Origination errors can also include incidents that weren’t completely accidental, more commonly referred to as BGP hijacks. Perhaps the most famous BGP hijack was the incident in February 2008 involving the state telecom of Pakistan, PTCL, and YouTube. In that instance, the government of Pakistan ordered access to YouTube to be blocked in the country due to a video it deemed anti-Islamic.

Diagram of Pakistan Telecom hijack of YouTube in 2008 (source)

To implement the block, PTCL announced more-specific routes of YouTube’s BGP routes to intentionally hijack Pakistan’s traffic to the video streaming service. Once hijacked, PTCL’s goal was to black hole the traffic, preventing Pakistanis from being able to access Youtube. However, things went downhill when PTCL passed these routes to its international transit providers, who carried the routes around the world, blocking Youtube for a large portion of the global internet.

Since the PTCL-YouTube hijack, there have been other instances of localized traffic manipulation implemented in BGP leaking out to the internet. In 2017, Russian state telecom Rostelecom leaked out a curious set of routes including those from major financial institutions.

During both the internet crackdown following the military coup in Myanmar in 2021 and the Russian crackdown of social media following its invasion of Ukraine in 2022, telecoms in each of these countries attempted to block access to Twitter using a BGP hijack to black hole traffic. In each case, the intentionally hijacked BGP route was unintentionally propagated onto the internet affecting Twitter users outside of the originating countries.

Kentik visualization of Russian BGP hijack of Twitter in February 2022 (source)

In 2008, researchers outlined how BGP could be manipulated to conduct a man-in-the-middle attack over the internet. The first documented case of a BGP-based man-in-the-middle attack like the one outlined in 2008 was discovered in 2013, originating in Belarus and targeting the networks of major US credit card companies and governments worldwide.

Diagram of traffic misdirection due to BGP-based MITM in 2013 (source)

During a 6-day period in August 2013, spyware service provider Hacking Team conducted BGP hijacks on behalf of the Special Operations Group of the Italian National Military Police, according to leaked documents revealed during a breach of Hacking Team’s network.

And finally, in 2018, a security company Backconnect publicly defended a BGP hijack they admitted to performing in order to regain control of a botnet server responsible for DDoS attacks. Researchers subsequently found that the DDoS mitigation firm had been involved in numerous prior BGP hijacks and had been utilizing a DDoS-for-hire service to drum up business.

BGP AS path errors

Not all routing incidents involve the perpetrator specifying its own ASN as the origin of the erroneous route. In November 2018, MainOne, a large telecommunications company in Nigeria, leaked routes received from a number of its peers, including major content delivery networks, to its upstream transit providers.

One of MainOne’s transit providers, China Telecom, failed to filter these incoming erroneous announcements, integrated them into its own routing tables, and proceeded to propagate them onward to its many customers and peers. Consequently, a significant portion of internet traffic bound for the victim networks was misdirected through China. Shortly afterwards, MainOne confirmed the leak was caused by their mistaken router configuration. Despite the error, misdirected traffic could still have been subject to interception or manipulation.

In June 2019, Allegheny Technologies leaked thousands of routes learned from one transit provider (DQE Communications) to another, Verizon. The routes that Allegheny leaked included many more-specifics that had been generated by a route optimizer employed by DQE. The result was that these leaked more-specific routes propagated throughout the internet and misdirected substantial amounts of internet traffic to Allegheny, causing a severe disruption.

Diagram of the Allegheny Technologies BGP leak of June 2019 (source)

And finally, for a period lasting more than two years, China Telecom leaked routes from Verizon’s Asia-Pacific network that were learned through a common South Korean peer AS. The result was that a portion of internet traffic from around the world destined for Verizon Asia-Pacific was misdirected through mainland China. Without this leak, China Telecom would have only been in the path to Verizon Asia-Pacific for traffic originating from its customers in China. Additionally, for ten days in 2017, Verizon passed its US routes to China Telecom through the common South Korean peer causing a portion of US-to-US domestic internet traffic to be misdirected through mainland China.

Diagram of China Telecom leak of Verizon routes in 2017 (source)

In each of these incidents, the origins of the leaked routes were unaltered, meaning any BGP security mechanism based on verifying route origins would have had no effect.

Since the Allegheny leak involved more-specifics, RPKI Route Origin Validation (ROV) could have proved helpful had Verizon employed it for route filtering at the time of the leak. More-specifics have a longer prefix length and would have been rejected because they would have exceeded the maximum length set in RPKI, such as was the case for Cloudflare’s affected routes.

Overall, these types of leaks are harder to mitigate than origination leaks and can only be addressed by analyzing and filtering the AS paths of BGP routes. There are technical proposals such as Autonomous System Provider Authorization (ASPA) that are in discussion, but no internet-wide mechanism exists presently to eliminate these types of incidents.

Attacks on cryptocurrency services

This section focuses on BGP incidents (all of which were erroneous originations) which were intentional since they enabled larger attacks that successfully stole cryptocurrency, a particularly lucrative target.

In 2014, BGP hijacks were used to intercept unprotected communication between Bitcoin miners and mining pools. This allowed an adversary to obtain bitcoin that should have been allocated to the mining pool. While this incident serves as an example of a BGP hijack targeting behind-the-scenes communication of cryptocurrency mining, more recent attacks have used BGP to attack cryptocurrencies with a more direct approach: stealing currency from users of online cryptocurrency wallets.

In 2018, attackers employed a BGP hijack that redirected traffic to Amazon’s authoritative DNS service. Having hijacked the DNS traffic, the adversary answered DNS queries for the web-based cryptocurrency wallet “myetherwallet.com” with a malicious IP address. Users that received this erroneous DNS response were directed to an imposter “myetherwallet.com” website. Some users entered their login credentials which were then stolen by the adversary, along with the contents of their cryptocurrency wallets.

Diagram of the BGP and DNS hijacks targeting myetherwallet.com (source)

While more advanced DNS security measures (e.g., DNSSEC) could have prevented this attack, the primary protection in place was the Transport Layer Security (TLS) protocol, which requires all connections to be encrypted. When TLS establishes an encrypted connection, the server must present a valid certificate that vouches for the server’s identity. Because MyEtherWallet did use TLS, users that were directed to the imposter site were presented with a prominent warning that their connection might be under attack. Despite this, many users clicked past the warning, and the adversary amassed $17 million in the cryptocurrency Ethereum.

While the 2018 attack was quite effective and demonstrated the viability of BGP attacks against cryptocurrency at a large scale, there were some silver linings. In particular, communication was still (at least in theory) protected by the TLS protocol, which led to the security warning. In the majority of cases, the proper behavior for a TLS connection when it gets an untrusted certificate is to abort the connection, and newer versions of Firefox do not allow users to click past TLS certificate warnings. Additionally, had the website used DNSSEC to secure its DNS traffic, the attack would not have succeeded.

However, both of these security technologies were completely bypassed in an attack in 2022 on the Korean cryptocurrency exchange KLAYswap. As Henry Birge-Lee of Princeton described in his write-up, the attack on KLAYswap exploited several vulnerabilities of KLAYswap’s cryptocurrency exchange web app.

Diagram of KLAYswap attack (source)

The adversaries used BGP to hijack the IP address of a server that belonged to Kakao Corp and was hosting a specific piece of javascript code used by the KLAYswap platform. The adversary’s objective was to serve a malicious version of this code file that would ultimately cause users of the KLAYswap platform to unknowingly transfer their cryptocurrency to the adversary’s account.

However, like MyEtherWallet, KLAYswap, and Kakao Corp were using TLS, so without the adversary presenting a valid certificate to complete the TLS connection, the adversary’s code would not be loaded. This did not stop the adversary, as it used an attack known in the research community where, after launching the initial attack, it approached a trusted certificate authority (or CA, the entities that sign TLS certificates) and requested a certificate for the domain name of Kakao Corp’s server that was hosting the javascript file.

CAs have to operate under guidelines designed to prevent the issuance of malicious certificates, which require the CA to verify the party requesting the certificate has control of the domain names in the certificate. One of the approved verification methods involves contacting the server at the domain through an unencrypted HTTP connection and verifying the presence of a specific piece of content requested by the CA. This cannot be done over an encrypted and authenticated connection, as the party requesting the certificate may be requesting a certificate for the first time.

During the attack, when the CA went to verify the domain ownership, its request was routed to the adversary’s server because of the BGP hijack. This falsely led the CA to believe the adversary was the legitimate owner of the domain and caused it to issue a certificate to the adversary. The adversary then completed the attack by using this certificate to establish an “authenticated” connection with KLAYswap users and serve its malicious code. Ultimately $2 million dollars were stolen from KLAYswap users over the span of several hours.

This attack is particularly notable because it involves a BGP attack successfully exploiting a system that was compliant with current best security practices. Even more aggressive application-layer defenses like DNSSEC and better TLS certificate error behavior would have been ineffective at preventing this attack because the adversary did not manipulate any DNS responses and served its malicious code over a trusted encrypted connection. In the current web ecosystem, millions of other websites, including those following best practices, are vulnerable to this type of attack.

In August 2022, cryptocurrency service Celer Bridge was attacked using a BGP hijack that employed fake entries in AltDB, a free alternative to the IRR databases as well as forged BGP announcements. By surreptitiously altering the contents of AltDB, the attacker was able to trick a transit provider into believing that a small hosting center in the UK was allowed to transit address space belonging to Amazon Web Services, which hosted Celer Bridge infrastructure. The attacker then forged the AS path of its hijack announcements to include an Amazon ASN as the origin, thereby defeating RPKI ROV. The hijack enabled the attacker to redirect cryptocurrency funds to an account controlled by the attacker.

IP Squatting

The discussion above has focused mainly on the disruptions or security implications of misrouting IP addresses which were actively in use (i.e., routed) at the time of the leak or hijack. However, there are bad actors that announce normally unrouted IP address ranges that don’t belong to them for the purpose of evading IP-based blocklists and complicating attribution. This phenomenon is generally referred to as “IP squatting,” but since it involves unauthorized BGP announcements, it sometimes is also referred to as BGP hijacking.

Since there is no effective legal or technical measure preventing this practice, bad actors can announce previously unused IP ranges belonging to others until networks on the internet take steps to block them for this bad behavior. In July 2018, a network that became known as the “BGP hijack factory” was removed from the internet through a collective effort. However, such a remediation is highly unusual and cannot be counted on to keep the practice at bay.

Closing thoughts

Originally composed for the BITAG report on routing security, the preceding paragraphs discuss only the most notable of many incidents, accidental or otherwise, involving BGP over the years. This extensive list of incidents bolsters the case that networks must take routing security seriously and implement measures to either protect themselves or other parts of the internet.

At a minimum, we recommend using a BGP monitoring solution to make sure you are alerted when an incident such as the ones above affect IP address space belonging to your business or organization. Additionally, we recommend deploying RPKI ROV by both creating ROAs for your IP address space as well as configuring your routers to reject RPKI-invalid routes.

Additional recommended actions for routing security can be found on the website of Mutually Agreed Norms for Routing Security (MANRS), which describes itself as a “global initiative that helps reduce the most common routing threats.”

We have made tremendous progress over the last decade. For example, we have not experienced a large-scale origination leak in many years, and that is not an accident. Many engineers at many companies have worked to improve overall routing hygiene, and we are all the beneficiaries of such work. However, there is much more to be done before we can say we’ve secured the BGP routing protocol, so we must continue to make progress on this complex and difficult task.