Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing Is Hard (2019)

chriskiehl.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: Event Sourcing Is Hard (2019)

#111

Post by bob1029 »

hinkley wrote: post #110
akiselev wrote: post #19

I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparin…

Am I crazy for wanting a standardized WAL format that can be treated as an event stream for anything from replicas to search services to OLAP? Why can't we drink straight from the spigot instead of adding abstractions on abstractions?

Its honestly not that hard to build a WAL-style solution exactly the way you want it for your own application. You just have to get away from the "you shouldn't write your own xyz" boogeyman experience long enough to figure it out.

Do you know how to model your events using a type system?

Can you be bothered to implement a Serialize and Deserialize method for each event type, or simply use a JSON serializer?

Do you know how to make your favorite language compress and uncompress things to disk on a streaming basis?

Do you know how to seek file streams and/or manage a collection of them all at once?

Can you manage small caches of things in memory using Dictionary and friends?

Are you interested in the exotic performance possibilities that open up to those who leverage the ring buffer and serialized busy wait consumer?

If you responded yes to most or all of the above, you are now officially granted permission to implement your own WAL/event source/streaming magic unicorn solutions.

Seriously, this stuff is really easy to play with. If you follow the rules, you would actually have a really hard time fucking it up. Even if you do, no one gets hurt. The hard part happens after you get the events to disk. Recovery, snapshots, garbage collection - that's where the pain kicks in. But, none of these areas is impossible. Recovery/Snapshots can again be handily defeated by the mighty JSON serializer if one is lazy enough to succumb to its power. Garbage collection can be a game of segmenting log files and slowly rewriting old events to the front of the log on a background thread. The nuance is in tuning all of these things in a way that makes the business + computer happy at the same time.

Plan to do it wrong like 15 times. Don't invest a whole lot into each attempt and you can pick this up really fast. Try to just write it all yourself. Aside from the base language libraries (file IO, threading, et. al), JSON serializer and GZIP, you really should just do it by hand because anything more complex is almost certainly wrong.

Re: Event Sourcing Is Hard (2019)

#112

Post by gunnarmorling »

akiselev wrote: post #19

I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparin…

Thanks a lot for mentioning Debezium. Agreed that a CDC-based approach will be simpler to implement and reason about than ES in many cases. Semantics are different a bit of course (e.g. ES events capturing "intend"); we had a post discussing the two things a while ago on the blog [1]. Speaking of capturing intend, we just added support for Postgres's pg_logical_emit_message() function in today's release of Debezium 1.8.0.Beta1 [2].

Disclaimer: I work on Debezium

[1] https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/

[2] https://debezium.io/blog/2021/11/30/debezium-1.8-beta1-relea...

Re: Event Sourcing Is Hard (2019)

#113

Post by gunnarmorling »

claytonjy wrote: post #59
bruth wrote: post #44

Earlier quoted context omitted.

I have not watched that clip, but as I said above, Confluent isn't a good resource for defining this term since Kafka cannot be used to do ES. I would suggest an article like: https://domaincentric.net/blog/eventstoredb-vs-kafka Also refer to the list of great resources @oskar_dudycz posted in another comment: https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...

that first article finds only one issue with Kafka-for-ES: limited number of partitions. Use pulsar, problem solved? Doesn't have the support or ecosystem of Kafka, but surely more than eventstoreDB.

There's two main reasons why Kafka as of today isn't a good fit for implementing event sourcing:

- Lacking support for key-based look-ups: when trying to materialize the current state of an entity, you cannot query Kafka efficiently for just those events (a partition-per-entity strategy doesn't scale due to the partion number limitation, which may change with KRaft in the future)

- Lacking support for (optimistic) locking: you cannot make sure to reject events based on outdated state, unless you apply work-arounds like a single writer architecture

Re: Event Sourcing Is Hard (2019)

#115

Post by Salgat »

adamauckland wrote: post #54

Event Sourcing is absolutely brilliant if you want to build an offline first/distributed system which will become eventually consistent. A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. I would abso…

Event Sourcing can be as consistent as any other data source. For a hotel booking company, I'd have each hotel be a unique aggregate stream, and with expected version assertion, a ReserveRoom Command for a room would only succeed if the latest state had an opening for that room. As soon as someone successfully reserves a room, all further reserve commands fail until another event frees up the room. The expected version assertion on writes guarantees this.

Re: Event Sourcing Is Hard (2019)

#116

Post by mixmastamyk »

What the hell is event sourcing? I read the first half of the article and gave up. Sounded like one of those distributed log things like kafka or maybe a message broker like rabbit?

Yes, could google, but now am discouraged. If you're going to write one of these, at least link to the wikipedia page:

https://en.wikipedia.org/wiki/Domain-driven_design#Event_sou...

Re: Event Sourcing Is Hard (2019)

#117

Post by jdmichal »

xwolfi wrote: post #75
mpfundstein wrote: post #58

Earlier quoted context omitted.

you sound like a consultant. few questions: - what was your biggest ES system that you worked on? - how many people worked on it? - how did the ES system particulary solve your problem? - what was the tech stack? Thanks

Not the OP, not a consultant, the biggest ES system I work on is an order management system in one of the biggest investment bank in APAC, which processes orders from clients to 13 stock exchanges. 40 people approx work on it in Asia, maybe around 100 globally at the raw dev level. I feel it's a sort of false good idea for our particular problems. Clients trade ether at the 10ms latency for high value orders or sub-m…

Are you really using event sourcing, or just message queues?

I worked for a large US bank, and had close contact with those who worked on the asset settlement system. I don't have as deep insight as you do, obviously. But the general architecture you describe sounds very similar. Except they clearly used message queues and treated them as message queues, and not event streams / sourcing / whatever. They used a specific commercial message broker that specializes in low latency.

Re: Event Sourcing Is Hard (2019)

#118

Post by akiselev »

bob1029 wrote: post #111
hinkley wrote: post #110

Earlier quoted context omitted.

Am I crazy for wanting a standardized WAL format that can be treated as an event stream for anything from replicas to search services to OLAP? Why can't we drink straight from the spigot instead of adding abstractions on abstractions?

Its honestly not that hard to build a WAL-style solution exactly the way you want it for your own application. You just have to get away from the "you shouldn't write your own xyz" boogeyman experience long enough to figure it out. Do you know how to model your events using a type system? Can you be bothered to implement a Serialize and Deserialize method for each event type, or simply use a JSON serializer? Do you k…

100% agree. I only used wal2json because my risk tolerance for the project was between "can't be bothered to pay the onboarding/maintenance cost of Kafka for Debezium" and "can't be bothered to implement a reader for the (well documented IIRC) stable binary WAL format" which is a weird spot to be in. This was a PoC written meant to demonstrate how we could implement the features we needed from ES using no more than standard Postgres tooling and a tiny service that was good enough to roll right into production with minor changes. It took under a week, though ideally I would have take the time to decoded the binary WAL format directly after saving the raw stream for safety's sake. Rust wasn't even an option at the time, nowadays it'd be mostly a bunch of macros and annotations with a sprinkingly of hand written FromBytes implementations and tiny bit of IO+serde glue code.

IIRC it took another data scientist and engineer under a month to turn the raw WAL logs into an audit log interface with pretty SSO avatars and weekly reporting. Someone from the devops team with DBA experience implemented time traveling staging DBs with continuous archiving and point in time recovery from production in the same time. Someone else later improved it so PITR used full backups created from a filtered WAL log so devs could select which parts of the production DB they copied over instead of each babying their own staging cluster that took days to rebuild. The whole project ended up giving us all of the benefits of event sourcing using standard, well tested tooling for a fraction of the cost.

We've thrown around the phrase "not invented here syndrome" so much that we've over corrected - as humans are wont to do - and now the younger generation thinks architectures like event sourcing or infrastructure like Kafka are a better solution than to just consume replication logs over a TCP connection to one of the most popular open source databases on the planet (not directed at the GP but my former coworkers :)). I'm starting to wonder if I've reached the age where I sound like the adults in the Peanuts cartoons, except the sound vaguely resembles "Get your resume driven development off my lawn!"

Re: Event Sourcing Is Hard (2019)

#119

Post by speed_spread »

hvgk wrote: post #101
tsss wrote: post #100

Earlier quoted context omitted.

You don't give people enough credit. I know damn well that the "complicated" solution is at best a monumental waste of money but doing the same trivial CRUD shit for years on end is neither intellectually stimulating nor good for my career. Until "the business" finds a way to change that I will use every available opportunity for "resume-driven development". Really, I don't give a crap if the shareholders make money…

Oh yes same here. I wrote a code generator that wrote all the code in the end. Now I maintain a shitty code generator. You can’t win. I think you eventually run out of fucks. I have.

That's when you write a fuck generator, so fucks can be given automatically on your behalf while you fuck off somewhere else.

Re: Event Sourcing Is Hard (2019)

#120

Post by acjohnson55 »

oskar_dudycz wrote: post #108
goostavos wrote: post #99

Earlier quoted context omitted.

Howdy! Author here ^_^ I'll respond to a few items because, even though I haven't touched the system in a few years, I could still rant endless about the mistakes I made building it. Deep scars were acquired! Firstly, to make sure we're talking about the same thing, where are you setting the bar for whether or not we can call something "Event Sourcing"? For instance, just to clarify, in our system events were indeed…

Thanks for answering! The bar is as I described. The events are the source of truth, only if you're using them in the write model as a basis for the state rehydration. If you're using materialised view, even though it's built based on events, then you outsourced the truth to other storage. If you're doing a pattern that you're just using events to build up the materialised view you use for the write model logic, that…

You're right that there is a distinction between event sourcing within a service and events as coupling between services. But as someone who architected a service that used event sourcing internally and did not use event streaming, I feel the author's points still stand. I also currently work on a very complex application built on event sourcing principles, and again, the same challenges are present.

The reality is that event sourcing brings a lot of challenges that have to be grappled with. It's a great paradigm, but it's not easier than in-place mutation. At least, not without a bunch of tooling.

Post reply on HN