Ask HN: Has anyone fully embraced an event-driven architecture?

It has allowed us to scale mainly in two ways: maximising parallelism with respect to CPU, and doing other works while waiting for an RPC call to return.

Event-driven architecture by nature is more parallel and efficient, but comes with a weaker consistency guarantee when it comes to the ordering of events coming from multiple parallel sources.

In my experience, people tend to fall prey to these pitfalls, and ended up resorting to inappropriate workaround such as global locks and ad-hoc retry mechanism. These are most commonly done when trying to aggregate works coming from concurrent producers or when needing to handle communication failures.

In fact, communication failures and downtimes are the most prominent problem in microservice particularly when you need your data to be inserted into multiple data sources in an atomic way.

This is an inherent issue in distributed systems and you have to think what's the atomic unit of data that you wish to insert, and design your system based on this hard constraint. Making the operations atomic, idempotent or revertable are some of the solutions you may want to investigate, but the moral of the story, is that you need to make sure these additional complexities are justified.

For us as a company, we decided on the event-driven architecture after knowing not just the benefit, but also the cost that I've outlined above.

For simpler applications that don't need to be a) real-time and b) handle crazy amount of loads, think small internal applications, small business ecommerce website, I would resort back to good old non-event-driven system since it's the more pragmatic option.

I've seen several companies building an event-driven architecture even when they know there's no way they would need to scale beyond serving several thousands of request per hour in the next two years. I think they would've been better off with a simpler, synchronous model.

The GUI worked in a single thread, but your whole program didn't need to do that.

Being able to copy a file and type at the same time was a big deal.

That's not correct. You just needed to create a separate thread for the file operation. For some programmers that was a big deal indeed, the same could be said for some programming tools. But that wasn't a general case at all.

There were some ugly things, like the way file operations were treated down the OS level, but it wasn't impossible to make your application responsive. It wasn't even difficult... if you knew how to do it.

IMO: It's over-complicated. They can ship a change to a microservice while insulating the other services from risk, but that's just kicking the can for technical debt.

What happens is that, if a service hasn't shipped in a few release cycles, when an update is made to that service, we often find latent bugs. Typically they are the kind of bugs that could be found with simple regression testing; but the company put too much effort into dividing its code into silos. (Basically, they spent a lot of time dealing with the boundaries between their microservies instead of just writing clean, testable code with decent regression suites.)

---

IMO: Don't get too hung up on microservices and events. Focus on writing simple, straightforward code that's easily testable. Make sure you have high unit test coverage and a useful regression suite. Only introduce "microservice" boundaries when there's natural divisions. (IE, one service makes more sense to write in Node.js, another makes more sense to write in C#; or one service should run in Azure and another should run in AWS.)

This, BTW, helped immensely in a previous job. When I worked for Syncplicity, a major Dropbox competitor, we started with a monolithic C# server for most server-side logic, but we had a microservice in AWS to handle uploads and downloads. This helped immensely, because we ended up allowing customers to host their own version of the upload / download server. It was a critical differentiator for us in the marketplace.

> do all consumers know how to deal with late-arriving events In my approach in architecting event driven systems, this question is meaningless. For me some of the core tenants (for me) of architecting these systems are

1. Message ordering should not matter 2. Message delays should not matter (no temporal coupling) 3. Message replays should have no side effects

This means that Service B should have no knowledge of what happens in Service A. If something in Service A fails and gets sent to Service A's DLQ, it should have no impact on Service B. This often involves rethinking business requirements and processes to account for workflows that get interrupted - I've yet to find an insurmountable issue.

If however, we are talking about a message broker trying to deliver a message from Service A to Service B, failing and forwarding to Service B's DLQ, then there is a very simple solution. Have your message broker deliver messages to queues. Service B then processes and handles messages from the queue rather than having to immediately handle all incoming messages from the broker. This protects against a lot of failure modes. The only thing(s) that arrive at Service B's DLQ are exceptions thrown internally in Service B