The migration to 400G/800G: Part I

Global data usage

Of course, at the heart of the changes are the global trends that are reshaping consumer expectations and demand for more and faster communications, such as:

• Explosive growth in social media traffic
• Rollout of 5G services, enabled by massive small cell densification
• Accelerating deployments of IoT and IIoT (Industrial IoT)
• A shift from traditional office-based work to remote options

Growth of hyperscale providers

Globally, true hyperscale data centers may number less than a dozen or so, but their impact on the overall data center landscaping is significant. According to recent research, the world spent a combined 1.25 billion years online in 2020 alone.¹ About 53% of that traffic passes through a hyperscale facility. ²

Hyperscale partnering with Multi-Tenant Data Center (MTDC / co-Location) facilities

As demand for more lower latency performance increases, hyperscale and cloud-scale providers work to extend their presence closer to the end user/end device. Many are partnering with MTDC or co-location data centers to locating their services at the so-called network “edge”³. When the edge is physically close, lower latency and network costs expand the value of new low-latency services. As a result, growth in the hyperscale arena is forcing MTDCs and co-location facilities to adapt their infrastructures and architectures to support the increased scale and traffic demands that are more typical of hyperscale data centers. At the same time, these largest of data centers must continue to be flexible to customer requests for cross-connections to cloud provider on-ramps.

Spine-leaf and fabric mesh networks

The need to support low-latency, high-availability, very high bandwidth applications is hardly limited to hyperscale and co-location data centers. All data center facilities must now rethink their ability to handle the rising demands of end users and stakeholders. In response, data center managers are rapidly moving toward more fiber-dense mesh fabric networks. The any-to-any connectivity, higher fiber-count backbone cables and new connectivity options enable network operators to support ever-higher lane speeds as they prepare to make the transition to 400 Gigabits per second⁴ (G).

Enabling artificial intelligence (AI) and machine learning (ML)

In addition, the larger data center providers, driven in part by IoT and smart city applications, are turning to AI and ML to help create and refine the data models that power near real-time compute capabilities at the edge. Besides having the potential to enable a new world of applications (think commercially viable self-driving cars), these technologies require massive data sets, often referred to as data lakes, and massive compute power within the data centers and large enough pipes to push the refined models to the edge when needed.⁵

As data center managers look to the horizon, the signs of a cloud-based evolution are everywhere.

More
high-performance virtualized servers

Higher
bandwidth and lower latency

Faster
switch-to-server connections

Higher
uplink/backbone speeds

Rapid
expansion capabilities

 
Within the cloud itself, the hardware is changing. Multiple disparate networks typical in a legacy data center have evolved to a more virtualized environment that uses pooled hardware resources and software-driven management. This virtualization is driving the need to route application access and activity in the fastest possible way, forcing many network managers to ask, “How do I design my infrastructure to support these cloud-first applications?”

The answer begins with enabling higher per-lane speeds. The progression from 25 to 50 to 100G and above is key to getting to 400G and beyond, and it has begun to replace the traditional 1/10G migration path. But there’s more to it than increasing lane speeds, a lot more. We have to dig a bit deeper.

In the data network, capacity is a matter of checks and balances among servers, switches and connectivity. Each pushes the other to be faster and less expensive, to efficiently track the demand produced by increased data sets, AI and ML. For years, switch technology was the primary bottleneck. With the introduction of Broadcom’s StrataXGS® Tomahawk® 3, data center managers can now boost switching and routing speeds to 12.8 Terabits/sec (Tb/s) and reduce their cost per port by 75 percent. Broadcom’s Tomahawk 4 switch chip, with a bandwidth of 25 Tb/s., provides the data center industry with more switching capability to stay ahead of those increasing AI and ML workloads. Today, this chip supports 64x 400G ports; but at 25.6Tb/s capacity, semiconductor technology is taking us down a path where in the future we could see 32x 800G ports on a single chip. 32, coincidentally, being the maximum number of QSFP-DD, or OSFP (800G transceivers) that can be presented at a 1U switch faceplate.

So now, the limiting factor is the CPU processing capability. Right? Wrong. Earlier this year, NVIDIA introduced its new Ampere chip for servers. It turns out, the processors used in gaming are perfect for handling the training and inference-based processing needed for AI and ML. According to NVIDIA, one Ampere-based machine can do the work of 120 Intel-powered servers.

Figure 3: Ethernet speeds

With switches and servers on schedule to support 400G and 800G by the time they’re needed, the pressure shifts to the physical layer to keep the network balanced. IEEE 802.3bs, approved in 2017, paved the way for 200G and 400G Ethernet. However, the IEEE has just now completed its bandwidth assessment regarding 800G and beyond. The IEEE has started a study group to identify the objectives for applications beyond 400G and, given the time required to develop and adopt new standards, we may already be falling behind. The industry is now working together introducing 800G and beginning to work toward 1.6T and beyond while improving the power and cost per bit.

Switching speeds are increasing as serializer/deserializer (SERDES) that provide the electrical I/O for the switching ASIC move from 10G, 25G, 50G. SERDES are expected to hit 100G once IEEE802.3ck becomes a ratified standard. Switch application-specific integrated circuits (ASICs) are also increasing the I/O port density (a.k.a. radix). Higher radix ASICs support more network device connections, offering the potential to eliminate a layer top-of-rack (ToR) switches. This, in turn, reduces the overall number of switches needed for a cloud network. (A data center with 100,000 servers can be supported with two levels of switching with a RADIX of 512.) The higher radix ASICs translate into lower CAPEX (less switches), lower OPEX (less energy required to power and cool fewer switches) and improved network performance through lower latencies.

Figure 4: Effects of higher Radix switches on switch bandwidth

Closely related to the increase in radix and switching speed is the move from a top-of-rack (ToR) topology to either middle-of-row (MoR) or end-of-row (EoR) configuration, and the benefit that structured cabling approach holds when facilitating the many connections between the in-row servers and the MoR/EoR switches. The ability to manage the large number of server attachments with greater efficiency is required to make use of new high-radix switches. This, in turn, requires new optic modules and structured cabling, such as those defined in the IEEE802.3cm standard. The IEEE802.3cm standard supports the benefits of pluggable transceivers for use with high-speed server network applications in large data centers defining eight host attachments to one QSFP-DD transceiver.

Figure 5: Architectures shifting from ToR to MoR/EoR

Just as the adoption of the QSFP28 form factor drove the adoption of 100G by offering high density and lower power consumption, the jump to 400G and 800G is being enabled by new transceiver form factors. The current SFP, SFP+ or QSFP+ optics are sufficient to enable 200G link speeds. However, making the jump to 400G will require doubling the density of the transceivers. No problem.

QSFP-Double Density (QSFP-DD⁷) and octal (2 times a quad) small form factor pluggable (OSFP⁸) Multi Source Agreements (MSAs) enable networks to double the number of electrical I/O connections to the ASIC. This not only allows summing more I/Os to reach higher aggregate speeds, it also allows the total number of ASIC I/O connections to reach the network.

The 1U switch form factor with 32 QSFP-DD ports matches 256 (32x8) ASIC I/Os. In this way, we can build high-speed links between switches (8*100 or 800G) but also have the ability to maintain the maximum number of connections when attaching servers.

Modulation schemes

Network engineers have long utilized non-return to zero (NRZ) modulation for 1G, 10G and 25G, using host-side forward error correction (FEC) to enable longer distance transmissions. To get from 40G to 100G, the industry simply turned to parallelization of the 10G/25G NRZ modulations, also utilizing host side FEC for the longer distances. When it comes to achieving speeds of 200G/400G and faster, new solutions are needed.

Figure 7: Higher-speed modulation schemes are used to enable 50G and 100G technologies

As a result, optical networking engineers have resorted to four-level pulse amplitude modulation (PAM4) to bring ultra-high bandwidth network architectures to fruition; PAM4 is the current solution for 400GPAM4. This is based in large measure on IEEE802.3, which has completed new Ethernet standards for rates up to 400G (802.3bs/cd/cu) for both multi-mode (MM) and single-mode (SM) applications. A variety of breakout options are available to accommodate diverse network topologies in large scale data centers.

More complex modulation schemes imply the need for an infrastructure that can provide better return loss and attenuation.

Predictions – OSFP vs QSFP-DD

Co-packaged optics (CPOs) provide an alternate path to 1.6T and 3.2T. But CPOs will need a new ecosystem that can move the optics closer to the switch ASICs to achieve the increased speeds while reducing power consumption. This track is being developed in the Optical Internetworking Forum (OIF). The OIF is now discussing the technologies that might be best suited to the “next rate,” with many arguing for a doubling to 200G. Other options include more lanes – perhaps 32, as some believe that more lanes and higher lane rates will eventually be needed to keep pace with network demand at an affordable network cost.

The only sure prediction is that the cabling infrastructure must have the built-in flexibility to support your future network topologies and link requirements. While astronomers have long held that “every photon counts” as network designers look to reduce the energy per bit to a few pJ/Bit⁹, conservation at every level is important. High-performance cabling will help reduce network overhead.

MPO connectors

Until recently, the primary method of connecting switches and servers within the data center involved cabling organized around 12- or 24-fibers, typically using MPO connectors. The introduction of octal technology (eight switch lanes per switch port) enables data centers to match the increased number of ASIC I/Os (currently 256 per switch ASIC) with optical ports. This yields the maximum number of I/Os available to connect servers or other devices.

The optical I/Os use connectors that are appropriate for the number of optical lanes used. A 400G transceiver may have a single duplex LC connector with a 400G optical I/O, it could also have 4 X 100G optical I/Os requiring 8 fibers. The MPO12 or perhaps 4 SN duplex connectors will fit within the transceiver case and provide the 8 fibers this application needs. Sixteen fibers are required to match 8 electrical and optical I/Os, preserving the radix of the switch ASIC. The optic ports can be either single mode or multimode depending on the distance the link is designed to support.

The electrical lane rate then determines the output capabilities of the optical interface. Table 1 shows examples of the 400G (50G X 8) module standards/possibilities.

Table 1: 400G Capacity QSFP-DD with 50G Electrical Lanes

When lane rates are doubled to 100G, the following optical interfaces become possible. At the time of writing, 100G lane rate standards (802.3 ck) have not been completed; however, early products are being released and many of these possibilities are in fact shipping. Table 2, presented at ECOC 2020 by J. Maki (Juniper), shows the early industry interest in the 800G modules.

Table 2: 800G Capacity QSFP-DD with 100G Electrical Lanes

Duplex connectors

Connector speed limits?

Connectors typically do not dictate speed, economics do. Optical technologies were initially developed and deployed by service providers who had the financial means and bandwidth demands to support their development, as well as the long-haul links that are most economically bridged using the fewest number of fibers. Today, most service providers prefer simplex or duplex connector technology paired with optical transport protocols that use single-fiber connector technologies like LC or SC.

However, these long-haul solutions can be too expensive, especially when there are hundreds or thousands of links and shorter link distances to traverse; both conditions are typical of a data center. Therefore, data centers often deploy parallel optics. Since parallel transceivers provide a lower cost per Gigabit, MPO-based connectivity is a good option over shorter distances. Thus, connector choices today are not driven so much by speed, but by the number of data lanes they can support, the space they take up, and the price impact on transceivers and switch technologies.

In the final analysis, the range of optical transceivers and optical connectors is expanding, driven by a wide variety of network designs. Hyperscale data centers may choose to implement a very custom optical design; given the scale of these market movers, standards bodies and OEMs often respond by developing new standards and market opportunities. As a result, investment and scale lead the industry in new directions and cabling designs evolve to support these new requirements.

Additional helpful DC resources

At first glance, the field of potential infrastructure partners vying for your business seems pretty crowded. There’s no shortage of providers willing to sell you fiber and connectivity. But as you look closer and consider what’s critical to the long-term success of your network, the choices begin to narrow. That’s because it takes more than fiber and connectivity to fuel your network’s evolution...a lot more. That’s where CommScope stands out.