One of the objectives of the original “Computer Networks: A Systems Approach” book was to treat networking not as a series of artifacts that are handed down from on high, but rather as a design space to be explored. Jon Postel famously remarked that much of what the IETF was doing in the 1990s was filling in all the gaps in the design space that had been carefully avoided by the original designers of the Internet. MPLS was such a situation: adding a different sort of forwarding mechanism to the quite successful one on which the Internet was built, which probably accounts for much of the controversy that attended its development. So this week we’re looking back at the history of that particular design space exploration, at which we had a ringside seat.
I find your recent blogs, podcasts and musings on social media very insightful and refreshingly honest. Like someone who has nothing to prove, and lots of introspection. Very inspiring for the not-at-all-famous people like me 😀 Please keep them coming!
In this blog, I took 2 takeaways:
1. MPLS, like is often the case in tech, was initially a solution looking for a problem, and fortunately found a problem from the carrier’s desire to simplify operations for enterprise VPNs
2. Your superpower is explaining complex concepts in a simple manner, and you’ve repeatedly capitalized on it, like you said in this post: “Yakov and I later co-authored a book on MPLS which was an excellent career move on my part.” Thanks for sharing such honest introspection with your readers!
Nice article! I recall some bits of this. What about Ipisilon? Didn't they have a predecessor tag switching system? I recall they generated so much press that cisco had to respond, and MPLS became pretty important.
I remember many a push early on, championing the enterprise side of the equation. MPLS CE-PE iGP and the ask to include EIGRP (got a funny story on that but not for here) in the mix, the need for native ipmc support, better granularity on qos/rsvp and many other. The memories, the first implementations at scale, the fantastic convo's with you, Yakhov, Jim, Jeff, Eric, Ivan, Joel, Chris, Russ and many many more....
The title and first sentence really sucked me into your article. Was MPLS necessary? Do we treat networking from first principles or as a craft?
I should start by saying that I am a fan of "Patterns in Network Architecture: A Return to Fundamentals." I've found that a very enlightening text and I believe that the networking it describes would be an interesting and elegant alternate reality - albeit one we will never see widely adopted. It did convince me that networking is a math and not a science: meaning it exists as a symbolic system that is not related to physical reality. I think this makes Networking more powerful, because, like math, a given problem can be approached, modeled, abstracted and solved in many ways.
Enter MPLS... The folks who think MPLS was an unnecessary forwarding type probably had no clue how the majority of networks were built at the time. IP networks were typically implemented over an underlying network that provided traffic engineering and high availability for the WAN links. The problems in the early 2000s was that IP networks, including parts of the Internet and early broadband deployments, were being built by carriers as a the data payload on top of existing ATM and Frame Relay Networks - and growing at amazing rates. The link speeds were to the point that using 53B cells was both unnecessary and costly. Remember the SAR-tax? Over time, more of the payload of ATM networks became IP and another problem emerged: the mismatch of QoS definitions between the payload and the underlying transport. CBR and UBR could be mapped reasonably well to EF and DE, but I'm not sure anyone ever got any good use of ABR, VBR-RT, and VBR-NRT. However, the ATM networks did one thing really well, they provided a highly available, connection-oriented, traffic-engineered transport system to underly IP and also other networks. MPLS TE was one of the first applications, and important because it could be used as the underlying network instead of ATM, and it could be provided by the same physical box as the IP router.
MPLS also had a major economic driver. Cisco was becoming a large and important player, and wanted to expand their business more strongly into the carrier networking space. However, at this time they had no carrier ATM or FR equipment, and existing routers (e.g. 7200) were typically considered inadequate to be used directly without one of those L2s as an underlying transport network. These were the days of the lollipop router, or the router-on-a-stick. MPLS was initially promoted as something that either ATM or IP vendors could grow to support, and the carriers bought-in mentally and economically. Cisco sold a lot more gear, and then that gear became a lot more capable. There was a jump in the sophistication of router architecture and in top-of-the-line products. All that then started the consolidation of core networking technologies. This might be one reason for the cynicism you mention about MPLS. In many ways it seemed a planned and well executed way to displace other types of carrier core network equipment with Cisco (and eventually also Juniper) gear. Once L2-VPNs & VPLS had been developed over MPLS, there really seemed to be no turning back. Unification of equipment, simplification of suppliers and unified operations / IT systems has made MPLS the go-to technology for carriers around the world today. Heck - even PSTN voice is largely carried over MPLS. An interesting thing, however, is that MPLS stayed largely in the carrier core and edge, and never became a customer interface technology. The cynic might say that MPLS and the MP-BGP control plane are almost a carrier religion. Like ATM once was. One adopted by few others.
Which brings me to my point. I hope it was clear that I think MPLS was a well designed and well-backed technology that it was extremely important for over 2 decades and made the world a better place. Its problem has become what I think you were trying to say with the SD-WAN comment: that it has drifted into becoming (generally) a carrier craft with fewer advocates and less funded innovation. Today I think that there are more people and innovative research involved in technologies other than MPLS. Hyperscalers are eclipsing the spend and innovation of carriers, and leading technology choices and investment in forwarding the state of the art. Startups rarely work on carrier problems or MPLS because venture capital is not willing to invest in that space. The network world is being led by the hyper-scalers and they are building their own networks, components, and protocols. It's not right or wrong, just a different math. And at the end of the day, I think it speaks volumes to do 4 web searches. Search for "Google" and "Facebook" with the terms "MPLS" and then "SD-WAN." My point is one that I think David Meyer said best,
"
Engineering artifacts are no longer the source of sustainable advantage and/or innovation.
Perhaps surprisingly, the hyper-scale and open source communities have taught me that that actual artifacts (in our case network applications as well as HW/SW) are ephemeral entities and that the only source of sustainable advantage/innovation consists of:
- Engineering Systems (including tool chains)
- Culture
- People / Process
"
I believe this, and if I'm correct then neither MPLS, nor SD-WAN, is/was necessary. What is necessary is to espouse the engineering systems and toolchains that are in large use at the moment, innovative and open-minded culture, and to attract the best and brightest to our problem space, then enable them with reasonable process. IMO this also means we have to guard against defining a craft, and beware the dogma of a religion. FWIW, I think MPLS met these criteria two decades ago, and that "it's all good."
I find your recent blogs, podcasts and musings on social media very insightful and refreshingly honest. Like someone who has nothing to prove, and lots of introspection. Very inspiring for the not-at-all-famous people like me 😀 Please keep them coming!
In this blog, I took 2 takeaways:
1. MPLS, like is often the case in tech, was initially a solution looking for a problem, and fortunately found a problem from the carrier’s desire to simplify operations for enterprise VPNs
2. Your superpower is explaining complex concepts in a simple manner, and you’ve repeatedly capitalized on it, like you said in this post: “Yakov and I later co-authored a book on MPLS which was an excellent career move on my part.” Thanks for sharing such honest introspection with your readers!
Thanks Bhavesh!
Nice article! I recall some bits of this. What about Ipisilon? Didn't they have a predecessor tag switching system? I recall they generated so much press that cisco had to respond, and MPLS became pretty important.
I remember many a push early on, championing the enterprise side of the equation. MPLS CE-PE iGP and the ask to include EIGRP (got a funny story on that but not for here) in the mix, the need for native ipmc support, better granularity on qos/rsvp and many other. The memories, the first implementations at scale, the fantastic convo's with you, Yakhov, Jim, Jeff, Eric, Ivan, Joel, Chris, Russ and many many more....
The title and first sentence really sucked me into your article. Was MPLS necessary? Do we treat networking from first principles or as a craft?
I should start by saying that I am a fan of "Patterns in Network Architecture: A Return to Fundamentals." I've found that a very enlightening text and I believe that the networking it describes would be an interesting and elegant alternate reality - albeit one we will never see widely adopted. It did convince me that networking is a math and not a science: meaning it exists as a symbolic system that is not related to physical reality. I think this makes Networking more powerful, because, like math, a given problem can be approached, modeled, abstracted and solved in many ways.
Enter MPLS... The folks who think MPLS was an unnecessary forwarding type probably had no clue how the majority of networks were built at the time. IP networks were typically implemented over an underlying network that provided traffic engineering and high availability for the WAN links. The problems in the early 2000s was that IP networks, including parts of the Internet and early broadband deployments, were being built by carriers as a the data payload on top of existing ATM and Frame Relay Networks - and growing at amazing rates. The link speeds were to the point that using 53B cells was both unnecessary and costly. Remember the SAR-tax? Over time, more of the payload of ATM networks became IP and another problem emerged: the mismatch of QoS definitions between the payload and the underlying transport. CBR and UBR could be mapped reasonably well to EF and DE, but I'm not sure anyone ever got any good use of ABR, VBR-RT, and VBR-NRT. However, the ATM networks did one thing really well, they provided a highly available, connection-oriented, traffic-engineered transport system to underly IP and also other networks. MPLS TE was one of the first applications, and important because it could be used as the underlying network instead of ATM, and it could be provided by the same physical box as the IP router.
MPLS also had a major economic driver. Cisco was becoming a large and important player, and wanted to expand their business more strongly into the carrier networking space. However, at this time they had no carrier ATM or FR equipment, and existing routers (e.g. 7200) were typically considered inadequate to be used directly without one of those L2s as an underlying transport network. These were the days of the lollipop router, or the router-on-a-stick. MPLS was initially promoted as something that either ATM or IP vendors could grow to support, and the carriers bought-in mentally and economically. Cisco sold a lot more gear, and then that gear became a lot more capable. There was a jump in the sophistication of router architecture and in top-of-the-line products. All that then started the consolidation of core networking technologies. This might be one reason for the cynicism you mention about MPLS. In many ways it seemed a planned and well executed way to displace other types of carrier core network equipment with Cisco (and eventually also Juniper) gear. Once L2-VPNs & VPLS had been developed over MPLS, there really seemed to be no turning back. Unification of equipment, simplification of suppliers and unified operations / IT systems has made MPLS the go-to technology for carriers around the world today. Heck - even PSTN voice is largely carried over MPLS. An interesting thing, however, is that MPLS stayed largely in the carrier core and edge, and never became a customer interface technology. The cynic might say that MPLS and the MP-BGP control plane are almost a carrier religion. Like ATM once was. One adopted by few others.
Which brings me to my point. I hope it was clear that I think MPLS was a well designed and well-backed technology that it was extremely important for over 2 decades and made the world a better place. Its problem has become what I think you were trying to say with the SD-WAN comment: that it has drifted into becoming (generally) a carrier craft with fewer advocates and less funded innovation. Today I think that there are more people and innovative research involved in technologies other than MPLS. Hyperscalers are eclipsing the spend and innovation of carriers, and leading technology choices and investment in forwarding the state of the art. Startups rarely work on carrier problems or MPLS because venture capital is not willing to invest in that space. The network world is being led by the hyper-scalers and they are building their own networks, components, and protocols. It's not right or wrong, just a different math. And at the end of the day, I think it speaks volumes to do 4 web searches. Search for "Google" and "Facebook" with the terms "MPLS" and then "SD-WAN." My point is one that I think David Meyer said best,
"
Engineering artifacts are no longer the source of sustainable advantage and/or innovation.
Perhaps surprisingly, the hyper-scale and open source communities have taught me that that actual artifacts (in our case network applications as well as HW/SW) are ephemeral entities and that the only source of sustainable advantage/innovation consists of:
- Engineering Systems (including tool chains)
- Culture
- People / Process
"
I believe this, and if I'm correct then neither MPLS, nor SD-WAN, is/was necessary. What is necessary is to espouse the engineering systems and toolchains that are in large use at the moment, innovative and open-minded culture, and to attract the best and brightest to our problem space, then enable them with reasonable process. IMO this also means we have to guard against defining a craft, and beware the dogma of a religion. FWIW, I think MPLS met these criteria two decades ago, and that "it's all good."