Live data from Hacker News

Event Sourcing Is Hard (2019)

chriskiehl.com

101–110 of 126 posts

Re: Event Sourcing Is Hard (2019)

#101
post #100
post #51

Earlier quoted context omitted.

This is a common problem I see across many things. We had a guy who spent two weeks writing a report script that collated data uploaded into S3 and wrote out another data lump into S3 after some post processing then sent this data lump to a guy via email. This entire thing had a bunch of lambda layers to pull in the pipeline, a build pipeline in jenkins, terraform to deploy it. The python script itself was about 200…

You don't give people enough credit. I know damn well that the "complicated" solution is at best a monumental waste of money but doing the same trivial CRUD shit for years on end is neither intellectually stimulating nor good for my career. Until "the business" finds a way to change that I will use every available opportunity for "resume-driven development". Really, I don't give a crap if the shareholders make money…

Oh yes same here.

I wrote a code generator that wrote all the code in the end. Now I maintain a shitty code generator.

You can’t win.

I think you eventually run out of fucks. I have.

Re: Event Sourcing Is Hard (2019)

#102
This article is great.

I architected an ES system once. It was absolutely the right call, but we had to work through nearly every one of those problems.

Regardless of whether you do it well or not, engineers will find it cumbersome to work with and think that it's overengineered.

Re: Event Sourcing Is Hard (2019)

#103

The big issue, to me, with concepts such as event sourcing is that they are fundamentally presented as software engineering concepts. You get a nice picture showcasing the overall structure and a story or two of how succesful this is, but we never discuss computer science matters. I want to know things such as "what safety properties does this system afford me?", "how is it abstractly modelled?", "what global invaria…

What is driving the adoption of an event sourced model other than a need for properties/guarantees that a reasonably normalized relational model can't provide? If the engineering team isn't talking about tradeoffs and the formal properties of the system, that seems like an indictment of the local engineering culture rather than anything to do with event sourcing.

Re: Event Sourcing Is Hard (2019)

#104
I work at Temporal[1], where we solve a bunch of these problems for you by cleanly abstracting the critical event sourcing bits under our SDKs. I think everyone who tries event sourcing eventually rolls their own framework or takes one of the shelf, and Temporal is the most battle tested one I've ever found.

The reason I posted this was that I found it helpful in prepping for a workshop I was giving today - recording is not up yet but slides are here https://docs.google.com/presentation/d/1mAPYXNmRdDKepXylB91F...

1: https://temporal.io

Re: Event Sourcing Is Hard (2019)

#105

As this article pops out again, I'd like to the point that although it may have some valid points, those points are not about Event Sourcing. What's expressed in the article is the Event Streaming or Event-Driven approach. So when events are not the source of truth etc. All of the event stores that I know supports strong consistency on appends, optimistic concurrency. Many guarantee global ordering. Some help in idem…

Howdy! Author here ^_^ I'll respond to a few items because, even though I haven't touched the system in a few years, I could still rant endless about the mistakes I made building it. Deep scars were acquired! Firstly, to make sure we're talking about the same thing, where are you setting the bar for whether or not we can call something "Event Sourcing"? For instance, just to clarify, in our system events were indeed…

As someone who's evaluating event sourcing for a real business use case, I really appreciated your article. I wish more people published their event sourcing war stories, rather than most resources being exclusively from consultants selling snake oil.

I think the point OP is making is actually the same as what you conveyed in your article--several of the worst pain points you experienced aren't inherent to Event Sourcing, they're due to a particular implementation of Event Sourcing. The problem is that that implementation is pretty much the canonical one sold by Martin Fowler et al.

I can think of a few specific ways in which event sourcing could be different than what you experienced:

1) Distributed computing is always hard. Event Sourcing is not a magical cure-all to distributed computing woes, but it's sold as that, and the result is that people go into a distributed event-driven architecture without being prepared for the engineering challenges that await. Eliminate distributed computing, and many problems simply go away.

2) Eventual consistency is always hard, but Event Sourcing doesn't inherently require eventual consistency, even for cached projections. If your projections are pure functions folded over the event stream, then caching and updating the materialized projections with strict consistency is straightforward. In the same transaction as inserting a new event, you run your projection function once on each of the materialized projections and the new event (`state = projection(oldState, event)`). This is only possible when you're not trying to create your projections in a distributed fashion, so see #1.

3) Language choice makes a big difference for reducing the cognitive overhead of updating projections. Your compiler should be able to tell you every place where you need to update a projection to account for a new event. Preferably, you should be able to define a hierarchy of events so that it only yells at you about places where the event is actually relevant.

Re: Event Sourcing Is Hard (2019)

#106
post #100
post #51

Earlier quoted context omitted.

This is a common problem I see across many things. We had a guy who spent two weeks writing a report script that collated data uploaded into S3 and wrote out another data lump into S3 after some post processing then sent this data lump to a guy via email. This entire thing had a bunch of lambda layers to pull in the pipeline, a build pipeline in jenkins, terraform to deploy it. The python script itself was about 200…

You don't give people enough credit. I know damn well that the "complicated" solution is at best a monumental waste of money but doing the same trivial CRUD shit for years on end is neither intellectually stimulating nor good for my career. Until "the business" finds a way to change that I will use every available opportunity for "resume-driven development". Really, I don't give a crap if the shareholders make money…

[deleted]

Re: Event Sourcing Is Hard (2019)

#107

I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparin…

> 99% of the time when ES sounds like a good idea, the answer is to just use Postgres.

I like some of the things in Event Sourcing very much.

However, whenever I ask "Why not just use PostgreSQL as the event store until I'm 5 orders of magnitude bigger?" I never seem to get a really good answer.

Re: Event Sourcing Is Hard (2019)

#108

As this article pops out again, I'd like to the point that although it may have some valid points, those points are not about Event Sourcing. What's expressed in the article is the Event Streaming or Event-Driven approach. So when events are not the source of truth etc. All of the event stores that I know supports strong consistency on appends, optimistic concurrency. Many guarantee global ordering. Some help in idem…

Howdy! Author here ^_^ I'll respond to a few items because, even though I haven't touched the system in a few years, I could still rant endless about the mistakes I made building it. Deep scars were acquired! Firstly, to make sure we're talking about the same thing, where are you setting the bar for whether or not we can call something "Event Sourcing"? For instance, just to clarify, in our system events were indeed…

Thanks for answering! The bar is as I described. The events are the source of truth, only if you're using them in the write model as a basis for the state rehydration. If you're using materialised view, even though it's built based on events, then you outsourced the truth to other storage. If you're doing a pattern that you're just using events to build up the materialised view you use for the write model logic, that can lead to using Event Streaming tools like Kafka, Pulsar, etc. And, as you said, this is a dead-end. Read more on what I mean by getting the state in Event Sourcing system: https://event-driven.io/en/how_to_get_the_current_entity_sta...

Of course, rebuilding a state from events each time you're processing command may sound dubious. Still, event stores (even those with relational DBs as backing storage) are optimised to read events quickly. Typically, reading event 100 events are not an issue for them. Also, for a typical LoB application. The temptation to use Snapshots is strong, but it should be treated as an optimisation when there is huge performance requirements. They may get out of sync with events. They may be stale, etc. I wrote about that in: https://www.eventstore.com/blog/snapshots-in-event-sourcing. The critical part is to keep streams short. That's again a difference between Event Streaming and Event Sourcing. In Kafka, you may not care how long your topic is, as it's just a pipe. Event stores are databases, so the more events you have, the worse. It impacts not only performance but also makes schema versioning harder (more on that: https://event-driven.io/en/how_to_do_event_versioning/).

It's easy to fall into the trap, as using a streaming solution is tempting. They promise a lot, but eventually, you may end up having issues as you described.

Still, that can be said on any technology. Event Sourcing has some dark parts. Also, it's essential to highlight them, but it's dangerous to present other tools and patterns issues related to ES.

Event Sourcing by itself doesn't directly relate to eventual consistency, type of storage, messaging, etc. Those are implementation details and tradeoffs we're chosing. Each storage solution has it's patterns and anti-patterns. Relational databases have normalisation. Document databases are denormalised. Key-value stores have strategies for key definition. Event stores also have their specifics. The most important is (as I mentioned) to take into the account the temporal aspect of streams, so keeping them short. I agree that there is a huge gap missing in knowledge sharing. I'm working on the article recently about that aspect. Here's a draft: https://github.com/EventStore/blog-articles/blob/closing-the....

I think that you highlighted well what can happen if you use the Event Streaming solution as a basis for the Event Sourcing solution. But unfortunately, the title and description state that it's about Event Sourcing, which is unfortunate, as it's highly misleading for people who are not aware of what Event Sourcing is and may suggest that it's much more complicated than it's in reality.

See also my other comments below in the thread, where I commented also in more details on other parts.

Re: Event Sourcing Is Hard (2019)

#109

Event Sourcing is absolutely brilliant if you want to build an offline first/distributed system which will become eventually consistent. A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. I would abso…

Couldn't agree more.

I've followed this rough pattern to build a rock-solid offline-first core for a personal application: https://flpvsk.com/blog/2019-07-20-offline-first-apps-event-... It's got some pain points; in particular it's quite difficult to evolve the schema once your app is out in the wild. But event sourcing is very amenable to TDD, and once all my tests were passing I had a nearly bulletproof solution for my current schema.

Strangely, offline-first functionality is usually not what event sourcing is pitched for. Maybe it's just a a legacy of its enterprise origins, but I often see it sold as a solution for enterprise applications where it's liable to cause more trouble than it's worth.

Re: Event Sourcing Is Hard (2019)

#110

I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparin…

Am I crazy for wanting a standardized WAL format that can be treated as an event stream for anything from replicas to search services to OLAP? Why can't we drink straight from the spigot instead of adding abstractions on abstractions?
Post reply on HN