Why Pub/Sub Isn’t Enough for Modern Apps

Pub/Sub is the most appropriate way of architecting the delivery side of event-driven architecture (especially for the web).

Chat notifications in Slack. Your Uber driver’s current position. Gone are the days where an app simply presented static data or invoked the occasional API; today’s users expect applications to be fully responsive—not just in terms of UI, but in terms of data too.

This shift in how data is used by applications is universal, but introducing live data to existing applications is not a trivial task. As developers tackle these new requirements, they’ll quickly encounter several realizations:

1.  features necessitates a shift to an event-driven architecture (and yes, event-driven architectures are the future)

2. Event-driven architectures mean a lot more data being sent and received between back-end & front-end

3. REST-based APIs are unable to provide all that data with the low latency required for real-time features

At this stage, many developers will look to a different architectural pattern: Pub/Sub. Pub/Sub as a model for streaming data is well understood; take events from a publisher (such as a driver location service) and distribute them to all interested subscribers (such as a mobile app). Whenever a new event is published, all subscribers are automatically notified and can handle the data appropriately. Because of the push-oriented nature (data is sent from the publisher to subscribers when it changes, rather than subscribers independently requesting updates), throughput is increased and latency reduced—hurrah! All our beleaguered developers need to do is to pick a Pub/Sub event-broker and move on with their lives!

Reality Strikes

Sadly, it’s never that simple. All of this holds true at a conceptual level—Pub/Sub is the most appropriate way of architecting the delivery side of an event-driven architecture (especially for the web). Where reality fails to meet our expectations, however, is in assuming that delivering events is the hardest part of the problem.

How we get data from A to B is a foundational concern. What I’ve observed from working with numerous companies trying to do this kind of transition, however, is that the far more difficult, and far more important question, is whether their event-driven architecture meets their other application needs.

The Plumbing Underneath Event Delivery

Let’s take a step back for a moment, and look at what the underlying technologies behind REST and Pub/Sub provide.

For REST APIs, we use HTTP requests, which give us:

Well-defined semantics for interacting with data (GET, POST, PUT, DELETE, etc.)

Several decades worth of infrastructure investment around scaling the performance of HTTP-based systems

An incredibly mature ecosystem of tools (from open-source frameworks to turn-key solutions) for defining, building, hosting, validating & testing APIs.

Now let’s compare that to WebSockets (other options exist, but I suspect a socket-based approach is by far the most common way to implement streaming Pub/Sub, at least for Web & Mobile):

The ability to send messages from one side of a connection to the other.

In our rush to move from polling to streaming, we’ve given up a lot. And crucially, most of what we’ve lost falls outside the definition of Pub/Sub. There are any number of “Pub/Sub” systems out there—but because the definition of Pub/Sub is purely focused on the basic semantics of producing or consuming data, there’s no guarantee that a given solution will provide all, or indeed any, of the other system qualities that a real-world application requires.

Beyond Pub/Sub

This is starting to sound quite academic, so let’s look at some concrete examples. All of these relate to that central question of ensuring that an event-driven architecture meets requirements beyond simply delivering events from A to B:

Security. Most API & Pub/Sub frameworks have clear support for standard authentication/authorization patterns (e.g. OAuth). What isn’t standardized, however, is how to handle changes to authentication. With HTTP requests, it’s simple; when a user’s privileges change, any subsequent requests can be rejected. But how does that work with connected streams in Pub/Sub?

Discoverability. The documentation and discoverability of available resources is a key issue, both for internal development and when exposing public interfaces. REST has numerous tools to aid in this; Pub/Sub, on the other hand, has no such standards. The AsyncAPI initiative is a major step in the right direction, but requires support from whatever framework you end up using.

Enrichment. Modern APIs are rarely a simple 1:1 mapping to specific resources. Filtering, merging, routing and other dynamic transformations are a key part of the largest API Management frameworks. Similarly, event transformation is well established for internal event systems (e.g. ksqlDB, Apache Spark) — but this has yet to reach the mainstream in Web & Mobile-focused Pub/Sub systems.

Reliability. Because of their stateless nature, REST APIs are innately tolerant of availability issues. If a given client loses its network connection briefly, it can simply poll once it reconnects. With Pub/Sub, we don’t have this luxury — when a client disconnects, the Pub/Sub system (again — no standards here!) has to decide how to handle this and whether that client can seamlessly reconnect. The back-end (specifically, the event broker) has to detect client disconnection, handle any event backlog, and ensure that reconnection is handled securely. This can have a major impact on the perceived responsiveness of your application.

Scalability. Another side-effect of HTTP requests being stateless is that REST APIs are almost trivial to scale horizontally. Pub/Sub, with its long-lived connections, is the polar opposite — specifically due to the additional responsibilities that the back-end must take on (see Reliability above). With Pub/Sub we have to rethink load-balancing, caching, and infrastructure deployment due to the fact that we can’t just throw more hardware at the problem — a lot of these concerns have to be addressed at the software level, which makes the choice of event broker a critical part of your scaling strategy.

In Summary

All of these examples hopefully illustrate the central point—that Pub/Sub, by itself, is not enough for applications adopting an event-driven architecture. For any kind of real-world application, delivering events is only the beginning—how to secure, scale, and enrich those events are far more valuable challenges to solve.

The second, slightly more subtle observation, is that this space—event-driven architectures for Web & Mobile applications—is still relatively immature. There are few defined standards for streaming APIs. GraphQL is rapidly gaining adoption (most recently being used by Microsoft to rebuild Teams), and how that fits alongside the more conventional Pub/Sub model remains to be seen. All of which makes this technology space very exciting!

ZippyOPS Provide consulting, implementation, and management services on DevOps, DevSecOps, Cloud, Automated Ops, Microservices, Infrastructure, DataOPS, and Security

Services offered by us: https://www.zippyops.com/services

Our Products: https://www.zippyops.com/products

Our Solutions: https://www.zippyops.com/solutions

For Demo, videos check out YouTube Playlist: 

https://www.youtube.com/watch?v=4FYvPooN_Tg&list=PLCJ3JpanNyCfXlHahZhYgJH9-rV6ouPro

If this seems interesting, please email us at [email protected] for a call.

Relevant Blogs: