Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing is Hard

chriskiehl.com

Post reply on HN
Print view View original

111–120 of 166 posts

Re: Event Sourcing is Hard

#111

Post by mbrock »

ploxiln wrote: post #24
6t6t6t6 wrote: post #13

Earlier quoted context omitted.

EventSourcing is not a Framework, but a concept. The idea is to store not the current state of your app, but the transitions (events) that derive into the current state. Think about how git stores your source code as a series of commits. In theory it is a beautiful idea; in the real world, it is hard to implement.

In fact, git stores a full snapshot of your entire repo with every commit. It does not store diffs from the previous commit. When you do "git show " it generates a diff from the parent commit on the fly. There's a huge optimization though: it uses a content-addressed blob store, where everything is referenced by the sha1 of its contents. So if a file's contents is exactly the same between two commits, it ends up usin…

Also directory structures are content-addressed, so if a commit changes nothing in a given directory, the commit data won't duplicate the listing of that directory.

Re: Event Sourcing is Hard

#112

Post by detaro »

fetbaffe wrote: post #107

It is worrying that a central figure to Event Sourcing & CQRS like Greg Young reduces the "framework" to a function, a pattern match & a left fold. Linked in the article https://youtu.be/LDW0QWie21s?t=1926

Why do you find that worrying?

I interpreted that section as "the core bits are easy, and the frameworks people have built don't really help you with the non-easy problems that come later, so they provide little value", which (assuming the statement about the frameworks is true) seems like a reasonable position?

Re: Event Sourcing is Hard

#113

Post by dm3 »

nicodjimenez wrote: post #36

Event sourcing is still the best paradigm for exposing valuable service data to an undefined number of services downstream, especially when there’s enough data to make querying painful for some use cases. They are like a NoSQL buffer between services and databases. Not always necessary but sometimes useful. Event sourcing brings its own complexities (eg Kafka clients) but it’s still better than having one huge shared…

This has nothing to do with event sourcing. What you're describing is event-driven integration. Rarely the events used to source the domain model should be the same as the ones used for the messaging integration. Unfortunately, due to the repetition of the term "event" the purpose gets confused most of the time. The "Data on the Outside versus Data on the Inside"[0] paper makes a good distinction even though it doesn't use the same terms.

[0]: http://cidrdb.org/cidr2005/papers/P12.pdf

Re: Event Sourcing is Hard

#115

Post by fetbaffe »

detaro wrote: post #112
fetbaffe wrote: post #107

It is worrying that a central figure to Event Sourcing & CQRS like Greg Young reduces the "framework" to a function, a pattern match & a left fold. Linked in the article https://youtu.be/LDW0QWie21s?t=1926

Why do you find that worrying? I interpreted that section as "the core bits are easy, and the frameworks people have built don't really help you with the non-easy problems that come later, so they provide little value", which (assuming the statement about the frameworks is true) seems like a reasonable position?

It is not how I interpreted it. What I heard was salesman pitching an expensive product as a bargain.

Even though frameworks have lots of drawbacks, I think it solves one problem really well, it gives you, the team, a direction.

Doing Event souring & CQRS correctly takes years of experience, this can be concluded by reading articles like this one or watching any of video by Greg Young.

In a sense the origin of this article stems from the notion that no framework is needed. I think that is a setup for disaster by selling developers the idea that this is easy, when it isn't.

In my experience frameworks have often learned me how not do things. Frameworks are condensed experiences that you don't need to learn yourself, someone else has already done the mistakes for you. This is a huge time saver & gives you, the developer, experience at a lower cost.

With that experience going frameworkless can then be achieved if necessary.

However my interpretation of it can be exaggerated due to the fact it was a short statement without much context.

Re: Event Sourcing is Hard

#116

Post by jpz »

bigbluedots wrote: post #109
jpz wrote: post #52

It is hard to separate DDD, Event Sourcing/CQRS - they seem to all be joined up concepts promoted by a small circle of people. I worked on a blockbuster project financed by a local billionaire in a Gulf State where the entire shebang was “mandated” by the CTO and a well-known Scala consultancy. Event sourcing, CQRS, with DDD to define architecture. Let me describe one simple issue that was almost intractably complex…

I might be misinterpreting your example, but wouldn't you have both the auth service and the preferences service listening for a 'new user' event (which would contain all the details needed by both systems), and both if them acting on it? Of course it would get more complex to handle error conditions, e.g what do you do if there is validation on the preferences that fails...

There are commands and events in CQRS/ES - where you need commands and success reported from multiple subsystems from front end interaction, now you need a transaction coordinator for your “saga”

Typically the front end needed to get a success message to say user created, or user creation rejected (e.g. backend checking on valid postcode, valid username, etc - rejections which would come from two separate services.)

Re: Event Sourcing is Hard

#117

Post by viraptor »

tomnipotent wrote: post #60
jsmeaton wrote: post #51

Earlier quoted context omitted.

What about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format. Redux often does not have to keep track of versions, because the event stream is consistent for that session.

> Now you’re having to push versions into your events Most distributed systems already do this, especially if they use Thrift/Protobuf etc. It's par for the course.

The protocol version only handles the version of the structure. It does not change when you change the meaning of the field. For example "this is still foo-index, but from now on it's an index in system X, not system Y". (Yeah, bad example, you could change the field name here, but sometimes it's not that clear)

Re: Event Sourcing is Hard

#118

Post by antofar »

healsjnr1 wrote: post #102
antofar wrote: post #97

I've worked in a couple of ES based trading system (matching engines and algo trading both) and it was definitely the way to go, especially on the equity side were you can restart or snapshot the system every day. In my startup, I've designed the core transactional system and it's serving us well for 2 years now. It comes with its own challenges and you need more senior devs than your average crud system hence you us…

I think this is the kind of pragmatism that is needed when taking about event sourcing. It is a _very_ advanced architectural pattern. There is no easy 'Event Sourcing made simple' way to use it. I think one of the reasons for this is that the systems people often describe when they talk event sourcing are actually _three_ different, but interrelated, architectural patterns: - event sourcing (build your models based…

Very good points. Cleary defined bounded contexts are the way to go and also having a clear distinction between internally and externally visible domain events. Other tricky points to get right how to distribute events and the relationship order (as in stream ordered, totally ordered, entity ordered) based on the scalability and "correctness" requirement. For instance in algo-trading is not uncommon to have totally ordered events throughout the system.

Re: Event Sourcing is Hard

#119

Post by coldtea »

olalonde wrote: post #19
AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

Things like Redux or Bitcoin are more or less event sourced systems. It's basically the idea of deriving your application state from a series of business events and storing those as a single source of truth. It's very appealing in theory but as the article explains, it's a bit more complicated in practice (e.g. dealing with consistency).

So somewhat akin to the Command pattern?

Re: Event Sourcing is Hard

#120

Post by justjico »

jacques_chester wrote: post #23
fagnerbrack wrote: post #22

First we need to agree on what "Event Sourcing" means. In my view you don't need to implement every single Event Sourcing pattern to have an "Event Sourced" system. Say you have a TODO List app (yes, pretty cliche but that's ok). In that TODO list you have a in which you post the state of the TODO list to the server. That state is stored in the database in the form of an event "TODO_LIST_SAVED". When you want to "rep…

Kafka-oriented streaming folks talk about stream-table duality; the idea that one form can be expressed as the other. There is usually a little lip service paid to this idea before some heavy hints that actually , the stream is the true reality are dropped. My own view is that there are dimensions for any data of interest, expressing some ability to show an evolution of it. Frequently that dimension is time, or can b…

I believe this is where CQRS plays nice with “event sourcing”. You write all your events in one model, but can read them in multiple ones, that is if you can tolerate some read latency... and most systems are usually ok with that.
Post reply on HN