Wired for Success Part 2: Ethernet - the Backbone of Modern Cloud Native and Real Time Communication

In the second installment of our blog series about Ethernet, we continue to explore its remarkable evolution. In my previous blog, “Wired for Success: How Did Ethernet Become the Backbone of Modern Connectivity,” I delved into Ethernet's history since its inception in 1973. Today, Ethernet stands as the most widely used networking technology. Initially designed to operate on a best-effort basis for delivering packets across a network using Carrier Sense Multiple Access with Collision Detection(CSMA/CD), Ethernet has significantly evolved. While its “old” CDMA/CD foundation remains, numerous enhancements have transformed Ethernet into a vital component for mission-critical applications and use cases such as industrial automation, mobile networks (4G/5G), and even in ruggedized environments such as in warships.

In today’s modern world, Ethernet has evolved to incorporate various applications and technologies. One notable example is Time Sensitive Networking (TSN), which enhances Ethernet to become a deterministic networking technology. TSN enables the synchronization of network elements and endpoints, such as switches and routers, to prioritize traffic classes and provide accountable delay (i.e. latency), and guaranteed bandwidth reservation. TSN is based on numerous international standards, with the key ones being IEEE 802.1AS (which defines the time synchronization method) and IEEE 802.1Qbv (which defines the time-sharing method). By integrating these standards with the Ethernet standard IEEE 802.3, punctuality is ensured, allowing for transmission within a given period while simultaneously accommodating a mix of other communication protocols.

Discussing TSN further, Ethernet’s usage in mission-critical networks and the telecom industry showcases its versatility. In the 1980s-1990s, Time Division Multiplexing (TDM) was the primary method for synchronizing voice services. However, Ethernet has now emerged as the de facto transport technology, utilizing Precision Time Protocol (PTP) - another standard IEEE 1588 and IEEE 1588v2 - to synchronize clocks throughout the network. Addressing latency has become essential, and PTP serves as the solution to this challenge.

To appreciate the significance of PTP, it's important to understand that we're dealing with timing accuracy in a range of hundreds of nanoseconds. This represents extremely tight timing requirements for certain applications and use cases. Maintaining precise timing is crucial for operating distributed systems at scale, ensuring that various operations remain synchronous. Additionally, precise timing is especially crucial when handling critical processes that govern infrastructure operations.

To learn more about this topic, read my blog “Are Your Servers in Sync? Is Network Timing the Synchronized Swimming of the Digital Age.”

An excellent example of such systems is found in the telecom industry’s mobile networks. Within the Radio Access Networks (RAN), oscillators must synchronize with the base stations on the correct allocated frequency. The synchronization process is vital to ensure seamless communication between mobile devices and base stations, as any deviation in frequency can cause signal interference and dropped calls. Ethernet is employed in mobile transport networks for both 4G and 5G, where PTP is becoming a key requirement due to the network being split into Front- Mid- and Backhaul network designs. Essentially, this provides a pure Ethernet packet-based connection from the Radio Unit (RU) to the first Point Of Presence (POP), which is the Cell Site Router (CSR) connecting to the X-Haul networks. The X-Haul network then offers connectivity to various sized data centers where 5G cloud-native applications reside as services, enabled by Ethernet-powered cloud data center fabrics connecting computes.

A new innovative approach, with the potential to fundamentally transform the mobile network business logic overnight, is driven by Software Defined Networks (SDN) and Network Function Virtualization (NFV). These technologies are the precursors to the cutting-edge 5G mobile networks that not only provide faster connectivity, but are also more powerful, scalable, and secure. Additionally, enable a fresh way of delivering services within telecom networks. Instead of relying on purpose-built hardware and proprietary black-box solutions as in the past, these networks employ more open standards and applications developed by large ecosystems and the open-source community. By adopting cloud-native technologies and adhering to DevOps principles, these networks are realized on compute resources connected with Ethernet in data centers of various sizes.

Like its predecessors, 5G networks are cellular networks. All 5G wireless devices in a cell connect to the Internet and telephone network via radio waves through a local antenna in the cell. These new networks boast higher download speeds, eventually up to 10 gigabits per second. In addition to being faster than existing networks, 5G offers higher bandwidth, enabling it to connect a greater number of devices and improve the quality of Internet services in crowded areas.  Naturally, Ethernet acts as the packet-based solution within 5G, accommodating all the essential containerized microservices required for 5G functionality.

Ethernet-based cloud data center fabrics come in various sizes, from small edge data center fabrics implemented as Layer 2 network infrastructures to truly scalable 3-stage and 5-stage large data center fabrics. These larger fabrics deployed as Layer 3 infrastructure with dozens or even hundreds of Ethernet switches connected in a spine and leaf architecture, also known as CLOS. The CLOS architecture has its origins in Charles Clos’ crossbar switches for telephone call switching, and it is composed of leaf and spine switches.

The most prevalent design for these cloud data center fabrics consists of Ethernet switches that use Virtual eXtensible LAN (VXLAN) network with Multiprotocol Border Gateway Protocol (MP-BGP) and an Ethernet Virtual Private Network (EVPN) control plane.

All Ethernet switches are deployed in pairs to provide dual-home redundant connectivity to computes and to other switches. The leaf switch pairs interconnect to form a cluster, providing redundancy for the attached computes. Border Leaf (BL) switches are also deployed in pairs, ensuring dual-home redundant connectivity to external Provider Edge (PE) routers and Internet, with the option to interconnect for additional redundancy. This presents yet another interesting use case in which Ethernet is serves as the foundation for cloud-native 5G mobile networks.

In my first blog, I provided some historical times for Ethernet, here are some more that are relevant to this blog:

• In the early 2000s, discussions around 10Gigabit Ethernet began, leading to the standardization of IEEE 802.3ae 10 Gigabit Ethernet in 2002.
• The Metro Ethernet Forum, an organization devoted to defining and promoting Ethernet carrier network services, was established in 2001
• 10GBASE-T, which outlined running 10 Gigabit Ethernet over twisted pair copper cabling, was standardized in 2006.
• In 2008, efforts to ruggedize Ethernet for data centers emerged, with the IEEE working on Data Center Bridging (DCB) and Shortest Path First lossless forwarding techniques, while the IETF defined TRILL for multiple active paths.
• In 2010, after forming a study group within the IEEE to investigate a leap in Ethernet speed to 100G. However, disagreements about speeds complicated the standard development process. Server vendors argued that they wouldn't need 100G adapters for years, and as a result, they requested a 40G Ethernet standard. Consequently, a joint 40/100G 802.3ba Ethernet standard was ratified in 2010.
• With the release of IEEE 802.11ac (Wi-Fi 5) in 2013, wireless access points reached speeds of 2 Gbit/s or 4 Gbit/s, exceeding the 1 Gbit/s IEEE 802.3ab 1000BASE-T wired Ethernet uplink. Although 10GBASE-T had been standardized in 2006, it had limitations regarding the supported distances for Cat5e cabling. As a result, there was a demand for an intermediate standard that could uplink the 2 Gbit/s and 4 Gbit/s speeds from wireless access points over existing cabling infrastructure. The development of the 2.5GBASE-T and 5GBASE-T standards allowed wireless access points to reach their maximum speeds without being constrained by the Ethernet uplink speeds over a single existing Cat5e cable, while also remaining compatible with newer Cat6 and Cat6a cabling.
• In 2013, the IEEE began a study group for 400G due to rising bandwidth demand from mobile devices and social networking. Terabit Ethernet (TbE) refers to Ethernet speeds beyond 100 Gigabit Ethernet. The 400 Gigabit Ethernet and 200 Gigabit Ethernet standards were developed by IEEE P802.3bs Task Force, utilizing technology broadly similar to that of 100 Gigabit Ethernet. The standards were approved in 2018.
• In 2014, a group of companies, including hyperscalers such as Google and Microsoft, formed the 25 Gigabit Ethernet Consortium to develop a more cost-effective standard for data center networks. The IEEE subsequently started working on the 25 Gigabit Ethernet (25GE) standard as part of the IEEE 802.3by project.
• In 2019, Communication Service Providers (CSPs) began deploying 5G networks, the fifth-generation technology standard for broadband cellular networks. Defined by 3GPP, 5G is the planned successor to 4G networks, which currently provide connectivity to a majority of mobile phones. 5G promises to be faster than existing mobile networks, as it offers higher bandwidth and can connect a greater number of devices. And remember, Ethernet is used as the packet-based solution within 5G networking.

Infrastructure is everywhere, and Ethernet is not only powering 5G data centers to provide the foundation for all cloud native 5G applications, but it has also become the ubiquitous network technology found everywhere. While 5G may not be the only network technology capable of meeting the diverse requirements arising from industrial applications and a wide array of other services, it complements other wired and wireless technologies rather than replacing them. This will likely result in 5G and Ethernet forming hybrid network infrastructures.

In upcoming installments of my Ethernet History blog series, I'll explore the phenomenon of Ethernet speeds increasing by an order of magnitude roughly every decade. Currently, in 2023, we're in the Terabit Ethernet era (400GE), which suggests that Petabit Ethernet might be within reach by 2050. This advancement could introduce various intriguing use cases, such as long-distance transport networks based on Ethernet. Keep an eye out for the forthcoming infographic and informative YouTube video we'll be releasing to commemorate this significant milestone and highlight Ethernet's lasting impact on our digital landscape. Join me on this thrilling journey as we delve into the fascinating history and promising future of Ethernet technology.