Ask HN: Has anyone fully embraced an event-driven architecture?

If you want to use event driven + microservices, first make sure microservices make sense. Event driven is just a cool way to tie monstrous collections of services together if you have to go down that road.

If you can build a monolith and satisfy the business objectives, you should almost certainly do it. With a monolith, entire classes of things you would want to discuss on hackernews (such as event-driven architectures) evaporate into nothingness.

If that's the case, then why don't we write entirely in terms of events as our base architecture? It isn't because we are ignorant of event-based processing, it is because we want the other types of systems in our lives. Transactions don't really exist; they are an abstraction we add to certain elements of the world that are behaviors they perform in response to certain types of incoming events. API calls don't really exist, they are a stereotyped pattern of an event making a call and event sent back, tied to the first via some ID (which may be a TCP socket), with a response, and no further activity on that ID for that event, which is an abstraction we added on top of events for a certain very common type of call.

Working directly with event-based systems is the software architectural equivalent of writing in assembler. Sometimes you have to do it because nothing else will do. However, you are dropping to a lower level, with all that implies, particularly the fact that you are now responsible for any of those nice properties that you want to enforce. Very similar to how if you want to use UDP, but you want some of the guarantees of TCP, you are now responsible for those guarantees. There's nothing wrong with that. It's just something you need to be aware of in making your decisions.

Being the foundational architecture everything else is based on, event-based systems can do anything that is possible to do, again, quite similar to how assembler is what can do anything the CPU can do. The other abstractions function by providing limits on the event flows in the system for their power, again just as a higher-level language like C, or perhaps even more clearly Rust, simply can not be used to generate all possible assembler instruction sequences, because the way they fundamentally work is to exclude sequences from the set of all possible sequences to only contain sequences maintaining certain properties. Event-based systems can implement API-call-type sequences internally. Event-based systems can implement transactions by manually implementing all the requisite limitations of events and event ordering and what things do in response to events. Etc.

But, if you have a system that needs any of those guarantees, it's kind of silly to start writing at an event-based level, only to have to painfully reconstruct the guarantees already available to you. As the Ancient Wisdom goes, "Any sufficiently complicated event-based system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of TCP." Most of these abstractions are around for a reason.

Where things go wrong is when the abstractions become detached from their underlying implementation in people's minds, and in their minds, become the base abstraction. Probably the single biggest instance in this space of that problem is treating "the API call" as "the fundamental abstraction". API calls are an incredibly useful abstraction, but, at the same time, a really terrible primitive to be the bottom of your system. To build the API abstraction out of event flows involves throwing away a lot of the capabilities of events. If an API call is what you need, and it is a very common need, that's a virtuous simplification, but when your needs exceed what an API call can do, you can really wreck up a design trying to implement an event system back on top of API calls. I've seen it at least twice in my career; one of the products I'm responsible for can almost literally be seen as a rewrite of a previous version that made that mistake in a manner that basically fatally killed it architecturally and replacing it with an event based system at the core... which I then immediately implement an API call layer on top of which mostly ran the system... but... right at the the critical place... didn't, and I could reach back down the stack and use the raw event-based system in the core for a few critical bits of functionality.

This is my "alternate point of view". Everything is already event-based, even when you can't see it. However, that doesn't mean it's a good idea to work at that level of abstraction all the time any more than the fact CPUs run assembler means we should always be working in raw assembly code. It is not necessary to use raw events everywhere. It is not a betrayal of good design to have some API calls in your system, or a centralized transactional database, or even TCP (which most notably adds "ordering" to events). It is, however, necessary for software architects to understand that the event-based system underneath is fundamental, and that they view the other additional abstractions as islands of functionality based on the event-based core of the world underneath, and not confuse those islands with the bedrock. Many systems may not even expose event-based functionality at a raw level anywhere, but if you keep this principle in mind, if a raw-event use case ever pops up, your system will quite likely be ready to handle it.

However, I am an advocate of the pattern and have seen it used successfully repeatedly. The largest scale as the data lead for a product maintained by 100-200 developers and several thousand transactions per second.

To answer your specific questions

>handling breaking schema changes or failures in an elegant way, and keeping engineers and other data consumers happy enough?

We did not allow for breaking schema changes. If there is a breaking change, it's a new event/topic. We used Kafka and every topic needed to have a compatibility scheme defined (see https://docs.confluent.io/platform/current/schema-registry/a...) to clarify what constitutes a breaking change. Even though some claim that producers and consumers can be fully decoupled, you will need to have a good idea who your consumers are and the time horizon of the data they consume. Application engineers are usually easier to keep happy than machine learning practitioners and other data consumers that want to consume events emitted over a long time period, potentially years.

> As a trivial example, everybody talks about dead-letter queues but nobody really explains how to handle messages that end up in one.

Dead letter queues are a tool you can use when the context demands it, applying it wholesale is likely creating too much overhead. But to provide you with a specific example. Some emitted events will be revenue impacting and depending on your setup, you actually want to use the events for financial reporting (careful! some more info later). In this specific use-case, if you can't process a record, the last thing you want to do is throw the message away. Somebody will need to have a look at these records, fix the cause and then either re-emit the records based on what you know about them from the header or fix the records in the DLQ. So think about the guarantees you need to provide and decide whether a DLQ makes sense for your use-case.

Some other thoughts and considerations.

- Topics more or less directly become analytics tables. Almost creating a unified view on your application's data otherwise difficult to create.

- How are the messages emitted. Are the messages emitted from the application logic? If so, what guarantees do you need? What happens if the app crashes (e.g. after a DB transaction happens and before the event was emitted). Depending on what you need, have a look at the transaction outbox pattern.

In my opinion there's also a time factor involved, for example you think there's a huge potential but zero actual clients today. In that case if you think a monolith will do the job the first 5 years of its life, build a monolith.