Event Sourcing is Hard • Hacker News Board

Re: Event Sourcing is Hard

#21

Post by jacques_chester » Mon, Feb 04, 2019, 2:37 AM UTC

ZenPsycho wrote: post #10

ralusek wrote: post #8

Here's my take on Event Sourcing: it's not particularly well defined what an "event" is. Did the event happen yet? Did it succeed? Which part? I didn't like having an event ledger say "comment created," where my application is then meant to consume this, handle validation, potentially fail on the db operation, etc. So here is what I do: I basically combine Event Sourcing architecture with CQRS. Whenever a client make…

Having it clear not just in your own head, but your entire team's head what an event is, quite important, especially to keep in mind what happens when you "play back" a log. Do transactional emails get sent out again? Do upstream services record the playback as duplicate events? How are transactions handled? What external state are you unknowingly depending on for that playback to produce the same result?

> Do transactional emails get sent out again? Do upstream services record the playback as duplicate events? How are transactions handled?

The narrow problem here is that Event Sourcing is temporal, but not bitemporal. Or rather, the different kinds of temporality are frequently muddled.

Event streaming in the Akidau/Google style is bitemporal, but mostly accidentally, as a side effect of distinguishing "event time" (fact time) and "processing time" (transaction time / belief time).

(Snodgrass later proposed tritemporal models, including a timeline for when a fact-belief was viewed, which I find both brilliant and slightly terrifying).

The problems of evolving data models, which is the third-order problem, is often hardest when you put a log or stream at the centre of your design. Databases using SQL, for example, struggle with first-order (current state) and second-order (historical state) evolution. But they do much better on third-order evolution, since DDL is built into every relational database.

Re: Event Sourcing is Hard

#22

Post by fagnerbrack » Mon, Feb 04, 2019, 2:37 AM UTC

First we need to agree on what "Event Sourcing" means. In my view you don't need to implement every single Event Sourcing pattern to have an "Event Sourced" system. Say you have a TODO List app (yes, pretty cliche but that's ok). In that TODO list you have a in which you post the state of the TODO list to the server. That state is stored in the database in the form of an event "TODO_LIST_SAVED". When you want to "replay" back, just list all the events from the DB in chronological order while filtering the ones the user has access and pick the last, then rebuild the HTML using that one event converted into the view model.

Kaboom, you have an event sourced system that doesn't even use a queue.

The problem with trashing the idea is that people have a bad experience with it, either by over-engineering and trying to apply all the solutions with tools or hand-made implementations instead of using a Lean approach, or storing events in a type of business model that doesn't even require a database.

"Event Sourcing is Hard" is a statement as true as "web development" is hard, or "distributed systems" is hard, or "API" is hard, "eventual consistency" is hard... yet, we build those things every day doing the best we can. In fact, anything, any practice, any technique, any architecture can be hard because software is hard. Even harder is to not over-engineer something that can be very simple.

Simplicity is hard.

Re: Event Sourcing is Hard

#23

Post by jacques_chester » Mon, Feb 04, 2019, 2:47 AM UTC

fagnerbrack wrote: post #22

First we need to agree on what "Event Sourcing" means. In my view you don't need to implement every single Event Sourcing pattern to have an "Event Sourced" system. Say you have a TODO List app (yes, pretty cliche but that's ok). In that TODO list you have a in which you post the state of the TODO list to the server. That state is stored in the database in the form of an event "TODO_LIST_SAVED". When you want to "rep…

Kafka-oriented streaming folks talk about stream-table duality; the idea that one form can be expressed as the other. There is usually a little lip service paid to this idea before some heavy hints that actually, the stream is the true reality are dropped.

My own view is that there are dimensions for any data of interest, expressing some ability to show an evolution of it. Frequently that dimension is time, or can be mapped onto time.

But neither the stream nor the table is the truest representation. The truest representation is whatever representation makes sense for the problem. Sometimes, I want to clone a git repo. Sometimes I want to see a diff. Sometimes I want to query a table. Sometimes I want a change data capture stream. Sometimes I want to upload a file. Sometimes I want a websocket sending clicks. Sometimes you need a freight truck. Sometimes you need a conveyor belt. Sometimes a photo. Sometimes a movie.

Sometimes I talk about space vs time using the equations for acceleration, or for velocity, or for distance. These are all reachable from each other via the "duality" of calculus, but none of them is the One Truest Formula.

And so it is for data. The representation that makes the most sense for that domain under those constraints is the one that is "truest".

Re: Event Sourcing is Hard

#24

Post by ploxiln » Mon, Feb 04, 2019, 2:52 AM UTC

6t6t6t6 wrote: post #13

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

EventSourcing is not a Framework, but a concept. The idea is to store not the current state of your app, but the transitions (events) that derive into the current state. Think about how git stores your source code as a series of commits. In theory it is a beautiful idea; in the real world, it is hard to implement.

In fact, git stores a full snapshot of your entire repo with every commit. It does not store diffs from the previous commit. When you do "git show " it generates a diff from the parent commit on the fly.

There's a huge optimization though: it uses a content-addressed blob store, where everything is referenced by the sha1 of its contents. So if a file's contents is exactly the same between two commits, it ends up using the same blob. They don't have to be sequential commits, or it could even be two different file paths in the same commit. Git doesn't care - it's a "dumb content tracker". If one character of a file is different, git stores a whole separate copy of the file for it. But every once in a while it packs all blobs into a single file, and compresses the whole thing at once, and the compression can take advantage of blobs which are very similar.

Re: Event Sourcing is Hard

#25

Post by linkmotif » Mon, Feb 04, 2019, 2:58 AM UTC

the_duke wrote: post #9

linkmotif wrote: post #5

Earlier quoted context omitted.

Event sourcing is REALLY hard to figure out how to do “right.” A lot of getting it right is modeling knowledge/experience, understanding your domain. That said, you can succeed at building your entire arch around it and once you do, it’s glorious. Kafka Streams makes the technical aspects easy once you figure out how to model correctly.

Out of curiosity, how do you deal with consistency guarantees across aggregates? (which is much more relevant when your whole architecture is ES) I realize this is highly domain dependent. Some will be much less affected than others. But it's another drawback not mentioned, because now you start to need sagas/managers that coordinate across services with commit/rollback patterns , conflict resolution., etc.

As zenpsycho said, all you get is eventual consistency across aggregates, if you’re talking about projection aggregates. As you say, for domain aggregates you can wire up transactions by writing your own 2PC on top of Kafka exactly once semantics.

I would recommend only to people who are really committed to the idea or know what they are doing and modeling. It took me a long time because I went from zero knowledge of Java/DDD. First I had to learn Java, then Kafka, then KS, but still I was lost. Learning the most basic DDD was enough for me and did the trick though. Writing a 2PC to coordinate across aggregates wasn’t pleasant but also wasn’t hard with KS. The hard part was learning all the other stuff and the modeling.

I think a well done ES framework based on Kafka streams would maybe be the first ES framework that would have a chance. The primitives in KS seem just right.

Re: Event Sourcing is Hard

#26

Post by 6t6t6t6 » Mon, Feb 04, 2019, 2:59 AM UTC

ploxiln wrote: post #24

6t6t6t6 wrote: post #13

Earlier quoted context omitted.

EventSourcing is not a Framework, but a concept. The idea is to store not the current state of your app, but the transitions (events) that derive into the current state. Think about how git stores your source code as a series of commits. In theory it is a beautiful idea; in the real world, it is hard to implement.

In fact, git stores a full snapshot of your entire repo with every commit. It does not store diffs from the previous commit. When you do "git show " it generates a diff from the parent commit on the fly. There's a huge optimization though: it uses a content-addressed blob store, where everything is referenced by the sha1 of its contents. So if a file's contents is exactly the same between two commits, it ends up usin…

Yeah... My comment was an oversimplification, but you get the point. :)

With event sourcing there's also the concept of snapshotting, btw.

Re: Event Sourcing is Hard

#27

Post by AndrewKemendo » Mon, Feb 04, 2019, 3:09 AM UTC

rwilson4 wrote: post #15

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

Events are the cornerstone of product analytics. If you want to understand what your users are doing on your platform, and to look for opportunities to improve the user experience, events are a big part of that.

I've been working with user event funnels for years, with tools like mixpanel and others but it seems like this is a build state tool for development workflow.

Re: Event Sourcing is Hard

#28

Post by acjohnson55 » Mon, Feb 04, 2019, 3:11 AM UTC

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

Here's a talk that largely introduced the concept to me, Turning the Database Inside Out: https://www.confluent.io/blog/turning-the-database-inside-ou...

Re: Event Sourcing is Hard

#29

Post by codebje » Mon, Feb 04, 2019, 3:12 AM UTC

6t6t6t6 wrote: post #13

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

EventSourcing is not a Framework, but a concept. The idea is to store not the current state of your app, but the transitions (events) that derive into the current state. Think about how git stores your source code as a series of commits. In theory it is a beautiful idea; in the real world, it is hard to implement.

Basic event sourcing is quite simple to implement. All the bells and whistles people sell alongside event sourcing are hard - whether you do event sourcing or not.

Your typical application presents a user interface based on data in a set of database tables (or equivalent), the user takes some action, the database tables get updated.

The equivalent event-sourced application presents a user interface based on data in a set of database tables (or equivalent), the user takes some action, the outcome of that action is written to one table, the other database tables get updated.

For git, where "or equivalent" is the working copy. You could easily imagine a source code management system similar to git, but without storing history - every commit and pull is a merge resulting in only the working copy, every push replaces the remote working copy with your working copy.

But man, wouldn't it suck to be limited to only understanding the most recent state of your source code…

Re: Event Sourcing is Hard

#30

Post by agentultra » Mon, Feb 04, 2019, 3:12 AM UTC

I've never read or heard that event sourcing was supposed to be all sunshine and rainbows.

Microsoft wrote about their adventures going down this path [0]. It's well worth the read if you're considering it for your project.

For a few features in my company's current platform we use event sourcing. We collect plenty of small data points over time that is aggregated into rows to provide users summary information of reams of operational data. We tried aggregating the data in queries and despite our best efforts to optimize our indices, tables, and queries there wasn't any way to compute it in any reasonable amount of time.

The pain points for us:

Our UI team wasn't in sync with how data flows in an event-based system. There's a lot of friction there. We're slowly updating the team on task-oriented user experiences and breaking down our UI components to transmit their commands directly. For now a lot of our control plane has to break up the huge form data we receive into commands and send back partial responses to the client. As we move forward and find better UI patterns this has improved.

The control plane controls some data models that are not event-sourced and are mutable. Our users tend to expect to be able to rename and delete objects in the system that our event-sourced models refer to. It caused some confusion when certain views in the application that are built from our projections wouldn't see updated name of the object they had just renamed. And so we ended up doing the event-sourcing no-no of emitting the CRUD events so that our projects could appear in the manner our users expected. This is partly because of the aforementioned problems with the UI team but is also a problem with event-sourced models referring to data that can mutate over time.

However it hasn't been a hellish experience either. I took the liberty of developing some models of our event-sourced infrastructure and features in TLA+. This has been helpful to ensure that certain properties of the system under development would hold: consistency, availability, etc. You may not be willing to go down the path of learning TLA+ but the key take-away there was that a little planning goes a long way with a project like this: simple unit tests and whiteboard diagrams are not going to cover all of the things that can go wrong in an event-sourced system. If anything it might convince you to keep it simple, limited, and constrained as we did.

edit: forgot link

[0] https://www.microsoft.com/en-ca/download/details.aspx?id=347...