Why APIs Matter
APIs (application programming interfaces) are no longer just a technical detail of interest to computer scientists. Banks, for example, are launching API platforms, and we hear the term “API first” thrown around in discussions of business strategy. We’ve used APIs for decades so we decided to take a look at why they are seemingly more important now than ever. We’ll have more to say on APIs in coming months.
In last month’s article about the big ideas powering the success of the Internet, we included the Socket API as one of the key contributors to the early growth of networked applications. It’s striking to remember that APIs (application programming interfaces) have been around for about as long as we’ve had programming languages, and that while the “API economy” might be a relatively recent term, APIs have been enabling innovation for decades.
Explaining what an API is can be surprisingly difficult. I like this short video from Martin Casado, which starts with the analogy of building cars. In the very early days, car manufacturers were vertically integrated businesses, essentially starting from iron ore and coal to make steel all the way through to producing the parts and then the assembled vehicle. As the business matured and grew in size, car manufacturers were able to buy components built by others, and entire companies could be created around supplying just a single component, such as a spring. This really gets to the heart of why APIs are important: they allow companies to access functionality supplied by others rather than having to build everything in-house. One of the most famous examples would be Stripe. By providing a complete suite of payment services consumable through an API, Stripe enables thousands of companies to build a business (e.g., an e-commerce site) that handles payments without writing the code to process payments themselves. In this telling, APIs are the technology that lets companies focus on their specialty, whatever that may be, and programmatically access the functionality they need in support of the business (payments, shipping, notifications, etc.) as services provided by others (who specialize in those supporting areas). Importantly, APIs are designed to be consumed by other software, easing the integration of a software component into a larger system.
I would argue that one of the most important developments leading to the central role that APIs play in technology and business today was the realization that APIs could be exposed as endpoints on the World Wide Web. Early efforts in this space included XML-RPC (the basis for SOAP) and the RESTful approach first proposed in the Ph.D. thesis of Roy Fielding. (Those of us who were around at the time remember the SOAP vs REST debates.) I consider this one of those inflection points that only looks inevitable with the benefit of hindsight. We take it for granted now that anyone can expose their service via an API that is accessed using HTTP, and that the tools necessary to both build and consume those APIs are widely available.
The world is now full of companies that are the software equivalent of the old automotive parts suppliers. Companies provide payment processing, shipping services, messaging, e-commerce platforms, all via APIs. For these companies, the API is the primary way that customers access their product. Sometimes it seems like the API is the product, since that is the part of the company that is the most visible–even appearing on billboards (at least in places like Silicon Valley).
Not only do APIs enable companies to focus on their primary business and leave supporting functions to others, it also enables them to offer services they might otherwise not be able to. For example, Stripe not only has expertise in detecting fraud, but they also have a breadth of historical data to feed into fraud detection algorithms that would never be available to a small or mid-sized company no matter their level of expertise. Companies can also offer additional services that go beyond their core business, e.g., an airline can offer travel insurance simply by calling the API of a specialized provider.
One early and famous example of an API that enabled a whole host of innovations is the Google Maps API. All sorts of data visualization tools were built on top of it (such as local crime rates, disease outbreaks, etc.), but beyond that, it enabled a generation of new applications, such as ride-sharing, that could not have existed without mapping and location services. So this was not just about enabling existing functionality to be handed off to a third party, it was also an enabler of new capabilities, much as the socket API fostered the growth of networked applications that had not been imagined previously.
I should emphasize that APIs are not just a way to access services from another company, but are an important tool for intra-company communication and collaboration as well. Jeff Bezos famously mandated the adoption of internal APIs at Amazon back in 2002, requiring that they be robust enough to support external users. While there is some debate as to whether that mandate has been truly embraced, the concept is sound and enables teams to develop their own functionality without destabilizing the work of others if they keep the API stable. It also laid the groundwork for AWS, which is essentially the result of making some of those internal APIs (and the services behind them) publicly available.
As a networking person, I’m intrigued by how the increasing importance of APIs also changes the way infrastructure is built and managed. The primary objects that we have to interconnect are no longer machines and routers but services that communicate via APIs. One salient example of this is the rise of service meshes: I think of a service mesh as the tool for managing policies, observability, access control, etc. for communication over APIs. Similarly, an API gateway (such as Kong) is an essential part of the infrastructure in this new world, providing services like authentication, load balancing, rate limiting and so on. Thomas Graf, founder of the Cilium project, summed it up nicely in the talk that first helped me understand service meshes: Layer 7 is the new Layer 4. (Cilium and eBPF are also highly relevant in this new way of managing infrastructure.)
The proliferation of APIs means we need tools to manage their complexity. With many tens of thousands of APIs available, how do you even find the right one? This has led to “API marketplaces”–a sort of app store for APIs (see RapidAPI for example). And we shouldn’t underestimate the challenges of integrating someone else's functionality into a complex system just because there is an API available. Unlike the lego bricks that often show up as an analogy when talking about APIs, there is a lack of standardization among APIs and connecting an API into a larger body of code takes more work than just “plugging it in”. There is a rich body of work to help with this complexity such as “connectors” that try to make the process a bit more lego-like.
As someone who has been dealing with APIs for about 40 years, I was initially puzzled as to why there is so much more focus on them now than when I was starting out my career. I think that the RESTful API development that started more than 20 years ago is one big reason, but I also like a point that I picked up from Martin’s automotive analogy. The car industry had to get to a certain size before it made sense to have companies build just one component of the car rather than the whole thing. Similarly, the software industry is so much bigger now, especially once you accept that software is a core part of most businesses (or eating the world, as Marc Andreesen said). So it stands to reason that, just as car companies no longer make their own steel, most companies will want to leverage APIs to bring in the best available functionality to enhance their products and services.
A great set of resources on APIs and how they are driving innovation in the tech world can be found at a16z. This podcast has some good analogies and gets at the question of how APIs matter to businesses. Our NFV article has been published in The Register. And we’re adding “Principles of Neural Design” to our reading list after a colleague pointed out the similarities between the Internet’s narrow waist and some of the design principles at work in the brain.
Great post! An interesting further point of discussion is the advent of "consumer data rights" which mandate companies (e.g. banks) making some of your personal personal data available via APIs.