5G and Data Center-Friendly Transmission Network Architectures
Introduction
In the last few years the demands from webscale network operators (WNOs, Figure 1) on transmission network architectures have changed considerably. From pure raw capacity requirements and lower costs, webscale players now prioritize highly scalable and advanced point to point bandwidth bundling interface technologies.
Figure 1: List of Webscale Network Operators (WNOs)
Webscale operators’ field of expertise is the data center, and most planned at least initially to rent capacity as leased lines from telcos (or telecommunications network operators, aka TNOs). However, many TNOs did not have the end to end transmission networks able to support webscale needs in terms of capacity, latency, cost objectives. Further, telco networks were not flexible enough to follow rapidly WNOs’ needs for modifications, additions, and changes of the services they needed.
Hence several years ago, WNOs themselves decided to build their own backbone and regional transmission networks, sometimes linking continents. Undersea, the WNOs either leased capacity from existing submarine consortia systems or started to build submarine cables for their own dedicated use. The largest WNOs, such as Microsoft, Facebook and Alphabet, have increasingly favored the latter (self-build) approach. With these initiatives, WNOs seek to have full control of the transmission network, and adequate time to market for their needs.
As webscale players have built out their networks, they have become more influential across the industry. Their buying power alone is a major reason; Figure 2 shows how webscale operator capex has grown dramatically since 2012, while telco capex has stagnated.
Figure 2: Capex – telco vs webscale (US$B)
At the same time, the telco market is far larger, and the largest integrated telcos spend well over 10% of capex on their transport networks. These telcos are heavily investing in the transformation of their transport networks, and supporting 5G is a central goal.
In the past, mobile services have been sold on the back of convenience of use. With very little considerations beyond coverage and without a capacity objective, there was never a firm commitment to service quality. 5G is probably the first access network with services subject to a wide variety of SLAs, from best effort to non-congested, and very low latency services with limits as low as 1ms. 5G will help operators to move from best effort services for all, to a tiered service level agreement (SLA)-based portfolio. Telcos hope this will help them to be more profitable, at least for the more sophisticated services.
Network slicing poised to play important role
The big change in direction in the strategy of transmission networks is that the planning, design, engineering and operations of a telecom operator will soon be subject to much tighter contracts and commitments.
For years, wild overbooking levels have been the norm, especially for mobile services, and networks were in most cases engineered for coverage alone. This won’t be possible for the next generation of services, which will require more than a 10x increase in bandwidth, and 10x less latency than the current generation.
In addition, webscale operators and large enterprises have demanding network KPI requirements. To serve this market, telcos must develop their transmission network end to end with enough flexibility to satisfy the capacity growth needs and resiliency requirements of these customers.
TNOs and WNOs both accept that the demanding requirements on bandwidth, latency and operational scalability to ensure short time to market for 5G services cannot be supported with existing network architectures.
A potential solution is “network slicing”, which starts with adding more TDM capabilities in the data plane to be able to provide a hard separation in the way services with different KPIs use network resources. This separation is orchestrated by an SDN centralized control and management plane.
Network slicing brings improvements to traffic engineering, with clear KPIs for bandwidth, latency and packet congestion. That helps to support all types of services over the same network infrastructure. Low priority services such as web browsing are effectively separated from network resources dedicated to services with demanding SLAs such as low latency leased lines or 5G inter-vehicle communications.
Operators pursuing FlexO technology to help cope with looming Shannon limit
Historically the main requirements telcos have standardized for transmission network architectures and platforms are high resiliency, powerful operations administration maintenance features, multiservice support, and backwards compatibility with legacy platforms.
This makes a lot of sense as most of the costs of running the network are operational in nature, such as repairs and maintenance. Further, multiservice capabilities can facilitate the migration of legacy services to newer platform. This reduces the need to support overlaid networks, and also avoids the cost of capacity expansions on older platforms at or near their end-of-life (EOL) dates.
In recent years, technological developments have pushed transmission networks towards the limit in the bandwidth per distance product, or the “Shannon limit”. The transmission technology is starting to hit the limits of the fiber medium.
One way to cope with this comes from the ITU, with its Flexible OTN, or FlexO, standard (G.709.1/Y.1331.1). FlexO allows client OTN handoffs above 100Gbps by defining an “OTUCn” modular structure: “an aggregate OTUCn (n ≥ 1) can be transferred using bonded FlexO short-reach interfaces as lower bandwidth elements.” FlexO also supports standard 100GbE optical modules.
FlexO has led telcos to consider how to fully exploit the flexibility of coherent transmission systems, allowing very high capacity transmission on non-regenerated short links, say 400Gbps links over 300Km distances, and lower capacity transmission over longer links, for example 100Gbps over 1500Km distances (figures for illustration only).
FlexO can bundle a number of lower rates at the TDM level to serve a higher capacity service for very long distances. For instance, by using inverse multiplexing or bundling a 400G service interface, capacity could be carried over four 100G links over (for instance) 1500kms without regeneration.
True to their backwards compatible requirements, telcos have made sure that FlexO supports 100G transmission requirements, and is an extension of existing OTN standards. This should simplify the roll out of FlexO on existing platforms.
FlexE to improve utilization, end-end manageability and router-transport connectivity
Operators – both telco & webscale – have also been exploring breakthroughs in the interfaces between transmission systems and servers and routers.
Aligned with FlexO, the Optical Internetworking Forum’s Flexible Ethernet (FlexE) supports similar schemes of bundling and multiplexing of interfaces between routers and transmission systems. FlexE offers a way to transport a range of Ethernet MAC rates whether or not they correspond to existing physical (Ethernet PHY) rates. Network utilization should improve, as should end-end manageability. One key element of FlexE was that Ethernet would grow within a TDM frame. This may pave the way to network slicing through the use of hard boundaries between tranches of services with different SLAs.
Most webscale operators lack an access link to the end user, making them rely heavily on telcos. And smaller webscale players like Netflix rent their clouds from other providers. Maintaining control of the user experience is an uphill battle. FlexO and FlexE help achieve this, in theory. On the UNI side, a WNO transmission network would now use FlexE interfaces with data platforms and servers and storage. On the NNI side, towards the fiber and other transmission systems, the WNO would use FlexO interfaces and standards.
Transmission interoperability improving due in part to the webscale push
Interoperability is something that transport engineers always wish for but never achieve due to network management interfaces’ lack of interoperability. Further, with coherent transmission, there is a problem with transmission interface incompatibility between vendors, each of whom can be more interested in higher performance and features differentiation than simplicity.
The telco response to the interoperability challenge has generally been to achieve subnetwork level interoperability rather than network element interoperability outright.
Things will change, though, as FlexO could be called the first optical standard that thrives on multivendor equipment operations.
Furthermore, webscale operators have designed simple transmission platforms and aimed to use cheap components already available from larger industries. Examples include the use of Ethernet interfaces components at 25G and 50G that were originally proposed for intra data center connectivity and rack cablings between servers and top rack unit switches. These will also be used in 5G base stations and mobile cloud engine platforms that require a transmission network to interconnect.
Conclusions
There is a growing alignment in the requirements for transmission network architectures across telecommunications and webscale network operators. They both need more flexible ways to grow their networks and manage them on an end to end basis. They want to benefit from low cost, open source components and procurement, but adapt technology to suit their customer base. They need to be able to support different classes of service and traffic. Even when providing free services, operators need to deliver a high quality of experience in order to monetize.
5G transport and data center interconnectivity services pose such a challenge to both TNOs and WNOs that work-arounds will not make up for limitations in either the data centers nor the network. For many operators, building a transmission network that supports network slicing principles will require a fresh start and new investments.
–
Source of cover image: CommScope.
[Note: a condensed version of this article first appeared at Telecomasia.net.]
