The Emerging Paradigm of the
Global Network Access Point (NAP)
PTC PUBLISH DATE: 4th MARCH 2021
President, InterGlobix & Executive Director, Internet Ecosystem Innovation Committee (IEIC)
Executive Vice President, Connectivity Strategy, QTS Data Centers
Network Access Point (NAP), Carrier Hotels, Cable Landing Station (CLS), Interconnection Ecosystem, Interconnectivity Hubs
THE DATA EXPLOSION
As the explosion of data continues to happen around us, the underlying infrastructure plays an ever- increasing pivotal role in moving the data from its source to the destination in a reliable, secure, and latency-sensitive way. In its journey the data (packets) could be traversing various types and numbers of networks including Wi-Fi, small and large cell networks, terrestrial fiber networks, and subsea fiber networks. The various networks interconnect with each other and exchange traffic mainly at a Network Access Point (NAP).
According to the research firm TeleGeography, the international Internet capacity growth in 2020 was 35%. Similar traffic growth statistics are reported from around the world from cloud providers, content distribution networks, Internet exchange points, and others. DE-CIX, which owns and operates the world’s largest neutral interconnection platform, saw a total of 32 exabytes of data traffic flow through its global platform in 2020 – roughly the same amount of data required for an 8eight-million-year-long video call or the amount of data that can be stored in 32 million laptops (assuming each laptop has 1TB hard drive).
A VITAL PART OF INTERNET INFRASTRUCTURE
The subsea cables carry above 99.7% of all international data traffic. It is estimated that the subsea cables represent USD 10 trillion worth of transactional value every day — five times the market cap of Apple and greater than the combined GDP of Japan, Germany, and Australia (Source: FCC).
To meet the exploding demand for data, the subsea cable industry has seen 107 new subsea cable builds, totaling a distance of 400,000 kms and USD 13.8B in value creation between 2016 to 2020. It took 162 years for 448 subsea cables and four years for 107 (Source: TeleGeography). This resurgence has been led by new players in the subsea sector (hyperscalers), and also newer (small) consortium models and faster deployment cycles.
HYPERSCALERS IN SUBSEA
The hyperscalers’ appetite for building data centers and leasing quality data center space continues to grow. In markets and situations where time-to-market is key, the leasing of quality space from trusted data center providers continues to expand. At the same time, the hyperscalers continue the land acquisition strategy to build and operate their own mega- campuses. Alongside this strategy, there has also been the foray of hyperscalers in subsea. In the last 10 years, we have seen multiple projects around the globe where hyperscalers have been investing in subsea infrastructure and capacity. This trend will continue.
At the end of 2020, approximately 34 subsea cable investments were made by hyperscalers, and that number continues to grow. These hyperscalers include Google, Microsoft, Facebook, and Amazon. Google owns approximately 102,362 kms of subsea cables (8.5% of all subsea cables); Facebook is at 91,859 kms; Amazon is at 30,557 kms; and Microsoft is at 6,605 kms (Source: BroadbandNow). By another count, over 60% of trans-Atlantic subsea capacity is now privately owned.
This has led to new partnerships, such as joint-build agreements of a small number of parties (one to three companies) and much faster deployment timeframes (less than two years for a trans-Atlantic system). Under the old consortium model, a large number of incumbent carriers used to invest in a shared subsea system, many times with projects lingering for years in the “agreement phase” before execution.
Going forward, large enterprises will likely become involved in subsea over the next five years following the lead of hyperscalers, as they build out their own private networks.
THE EVOLVING SUBSEA/CLS/DATA CENTER LANDSCAPE
The landing point of the subsea cables is the Cable Landing Station (CLS), which is commonly located near the beach where the subsea cable exits the water. The typical CLS houses the optical terminal equipment (SLTEs), the power feed equipment (PFEs), and provides the initial point of interconnection between the subsea and terrestrial networks.
With the advancements in optical technology, new systems can be designed with new express point-to-point architectures and even be significantly longer, while carrying more traffic per fiber pair. This has also allowed system designers to redesign the CLS, allowing only critical power equipment to be deployed close to the beach in small scale “PFE huts,” designed as lights-out shelters while moving the optical gear into the data center.
The end-to-end subsea design is driven by the needs of system owners and the availability of modern and diverse terrestrial backhaul facilities. In some cases – particularly hyperscalers – will want to optimize for a data center-to-data center design, while carriers may want to find points of interconnection or NAPs where they have access to a plethora of terrestrial network and content providers.
Multi-tenant data center providers have started to view subsea operators as strategic partners and vice versa. Subsea cables enable global connectivity and have become a catalyst in building network-dense, carrier-neutral NAPs. The subsea industry can realize the inherent benefits in terms of resiliency and redundancies built into the data center design, as well as the rich interconnection to terrestrial networks.
“While on one side, network route diversity is critical to increase service resiliency, on the other, interconnecting at a carrier-neutral, multi-tenant edge-colocation facility is as critical. Neutrality and clear rules of engagement are paramount to build trust among service providers and off-takers, and to further develop a solid interconnection ecosystem,” said Erick Contag, executive chairman of GlobeNet and president of the SubOptic Association.
“A highly reliable and scalable interconnection ecosystem tied to the global communications fabric is vital to support the ever-growing demand of data, online gaming, remote work, hybrid cloud instances, virtual environments, and other services that are the building blocks of our digital world.”
Executive Chairman of GlobeNet and
President of the SubOptic Association
THE GENESIS OF AN INTERCONNECTION ECOSYSTEM
Having open and neutral cross connect policies helps make the proposition stronger for various network parties to colocate in a carrier-neutral facility that has clear interconnection rules and facilitates open interconnection between various parties. From a subsea perspective, having the Submarine Line Terminal Equipment (SLTE) in the facility is a cost benefit, but it cannot always be done due to technical design limitations. When the SLTE is colocated in a carrier-neutral, multi-carrier data center with operational excellence, the subsea traffic can be directly interconnected to other network providers including terrestrial fiber providers (both dark and lit), Content Distribution Networks (CDNs), Multiple System Operators (MSOs) or “eyeball” networks, Software-Defined Networks (SDNs), cloud on-ramps, and general colocation customers. QTS Data Centers has taken this strategic approach to establish such interconnection-rich ecosystems or NAPs in Richmond, Virginia; Hillsboro, New Jersey; and
THE NEED FOR DIVERSITY
Within the United States, there are a handful (less than a dozen) of major Internet peering points or interconnection hubs. These hubs are generally known by their addresses as opposed to the builder owners or operators. This is where the majority of the Internet traffic exchange takes place. There are less than a dozen such interconnection hubs in the U.S. and probably as many (or less) across the world.
|21715 Filigree Court, Ashburn
|55 Marietta Street SW, Atlanta
|New York City
|111 Eighth Avenue, New York
|The Westing Building Exchange, 2001 Sixth Avenue, Seattle
|1950 North Stemmons Freeway, Dallas
|One Wilshire, 635 South Grand Avenue, Los Angeles
|NAP of the America (NOTA) 50 North East 9th Street, Miami
|350 East Cermak Road, Chicago
|200 Paul Avenue, San Francisco
|11 Great Oaks Boulevard, San Jose
As evident in the map below, if we look at the Eastern Seaboard pre-2017, most of the international subsea traffic landed either in the northeast (New York/New Jersey) with 13 subsea cables connecting to 10 countries, or in the southeast (Florida) with 22 subsea cables connecting to 34 countries.
Post Hurricane Sandy, Microsoft, Facebook, and Telxius formed a consortium to land a new subsea cable, MAREA (Spanish for “tide”), in a region completely diverse from the traditional northeast and southeast landing sites. They picked the mid-Atlantic region in the commonwealth of Virginia. This provided the much-needed diversity for subsea traffic coming from Europe. Facebook and Microsoft consumed the capacity for their internal requirements, and Telxius had the role of being the reseller of capacity, in addition to expanding and diversifying its own IP backbone. MAREA became active in the summer of 2017.
The following year, Telxius announced another subsea cable landing in Virginia, this time from South America. The cable connects Rio de Janeiro, Brazil, to Virginia with branching units along the way at Fortaleza, Brazil, and San Juan, Puerto Rico.
Most recently, the Google-owned Dunant subsea cable landed in Virginia with direct connectivity across the Atlantic to France. Orange and Telxius, being the landing parties in France and Virginia respectively, also have capacity on this system.
QTS RICHMOND NAP, PART OF A GLOBAL NAP INITIATIVE
QTS owns and operates the world’s fourth largest data center in Richmond, Virginia, which is about 95 miles inland from the beach where these subsea cables land. QTS was quick to recognize the subsea cables as a strategic initiative and stuck partnerships with the cable owners and consortium members to bring the capacity to the QTS Data Center.
This QTS facility in Richmond, over the last decade or so, went through several enhancements, upgrades, and carrier builds. With 16 network carriers, two SDNs, three Internet exchange points, and two cloud access networks, it proved to be an optimum location for the development of an interconnection ecosystem or a Network Access Point (NAP). With the international subsea capacity being brought to the NAP in Richmond, it led to the formation of the QTS Richmond NAP.
With the “open-access” policy, QTS levelled the playing field by providing access to the NAP to not just other carriers and network providers (all sub-sectors) but also to other data centers and enterprise locations. This has in turn led to the economic growth of the region and, in particular, at the White Oak Technology Park in Henrico, Virginia, where the NAP is located. Recently, DE-CIX, the global interconnection platform, announced DE-CIX Richmond, making the region part of North America’s largest data center-neutral and carrier-neutral interconnection platform and the world’s largest neutral interconnection platform.
As part of a strategic initiative, QTS has since announced other NAPs in various parts of the U.S. and the world including: QTS New Jersey NAP, QTS Hillsboro NAP, and QTS Eemshaven NAP (in the Netherlands). The interconnection ecosystem development is underway in each of these markets.
FORMATION OF IEIC
The success of QTS Richmond NAP led to the formation of the Internet Ecosystem Innovation Committee (IEIC), a global industry committee chaired by Dr. Vinton Cerf. Various industry leaders representing different sectors and companies have come together in support of a common mission – resiliency and diversity of the Internet.
The current founding members of IEIC are industry executives from Akamai, Bank of America, Blade, Ciena, Cigna, Google, Lumen, QTS, Subspace, Telxius, Virginia Commonwealth University, Hilton, InterGlobix, Uber, FreddieMac, and Ford. Additional information on IEIC can be found at www.ieicco.com.
As it is evident with the turn of recent events, our reliance on digital infrastructure will continue to increase at a rapid pace. The amount and rate of data growth will continue at an astronomical pace. The data center development will go hand-in-hand with the need for additional underlying infrastructure—both on land and at sea (and in some cases in the sky supporting new LEO satellite constellations).
The data center industry and subsea industry will continue to work together closely and more proactively. The industry will drive toward the formation of additional interconnection hubs globally and also diverse landing points for subsea cables.