Ask HN: Has anyone fully embraced an event-driven architecture?

"Everything looks like a red thumb when you're holding a golden hammer."

Events are a part of a greater whole. It's a tool that you can use to solve certain data flows, but not all data flows. When you start taking more liberty with the word "eventually," you are almost certainly in a realm where event-driven makes the most sense. CQRS is a pretty good example of using many architectures (including event-driven) under a single greater architectural umbrella, and the thought patterns it introduces you to are incredibly useful. But no architecture is gospel, not even close.

Any "pure" architecture is the tail wagging the dog. The problem comes first, the solution comes second, the architecture comes third.

I have at a few companies now.

It's great for processing data that goes beyond a single database call, data formatting and presenting something on a page.

If you're chaining multiple microservices together you've made a very sloppy/poor mans version of this. (People tend not to account for downed services, maintence updates, client durability, etc) When you bring in technologies like Kafka to orchestrate this, you'll end up with a more reliable system that you can fix if something goes wrong. This more changes the way you think about data and how you present it. Also, it'll increase your uptime because your service's SLA is isolated from what you're processing. (Your service and the persistent storage that is storing state is what people see.. the data being out of date is something you should account for)

Schema changes: Generally you don't have that big of a deal because multiple applications get started at once. You should have system level tests to catch that before you go out. Also, application smoke tests help as well. As long as you picked a durable message queue with a framework that'll crash on error, you can fix that, bring up the fix and continue processing through.

Dead letter queues: It's more about how you architect more than anything. This is something you should plan for.

Yea, I have use GitHub search in a pinch and sometimes it is helpful enough to show me exactly where to look. Unfortunately, though, there are several events we emit that are many layers of string concatenation, so GitHub search may narrow it down to 4 or so places and I have to manually go from there.

>What does manual intervention look like?

A ticket or an ops alert depending on severity. The cause gets fixed and messages are retried or dropped depending on your business case.

>Are all downstream consumers blocked until the DLQ is emptied

No, its a diffent queue. The point is that you eventually pull them into the DLQ so you aren't wasting all your resources on failing messages.

>do all consumers know how to deal with late-arriving events?

Hopefully, but an outage is an outage. You're way past happy path at that point.

I work in semiconductor manufacturing, it's a very common model. I have been in about eight fabs around the world that use it quite successfully.