Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing Is Hard (2019)

chriskiehl.com

Post reply on HN
Print view View original

31–40 of 126 posts

Re: Event Sourcing Is Hard (2019)

#31

Post by stepbeek »

kmac_ wrote: post #16

ES is really great, but just for a narrow set of service types and when it's applied locally in a closed bussines domain. But as a global system architecture, nope, I tried that once and it only brought more costs.

100% agree. I used it for position management software in equities and it fit like a glove. Effectively, we had a stream of transactions coming from upstream that we aggregated into a cache. Being able to replay events, or step through to figure out why something had gone wrong was terrific.

I then tried to apply this to a conventional request-response web service. It did not fit well. Everything was immediately significantly harder with only a marginal gain.

My rule fo thumb now is to use event sourcing when doing otherwise would be difficult. A justification based off of an improvement, rather than one solving an obvious problem, isn't worth the extra effort.

Re: Event Sourcing Is Hard (2019)

#32

Post by lucian1900 »

ddek wrote: post #25
justusw wrote: post #24

Is Event Sourcing compatible with GDPR?

The issue with GDPR is right to erasure, which is tricky in conventional systems. But in ES systems, naively implemented, it is not possible. If you are writing PII in plain text, an immutable log won't let you remove it. OTOH I have two options: store PII in a separate database, and just delete or nullify when requested; or use per-user encryption, and delete the key.

The GDPR concept is "personal data", partly since it doesn't just refer to ways to identify a person.

PII is a similar but different US concept.

Re: Event Sourcing Is Hard (2019)

#33

Post by tkubacki »

oweiler wrote: post #12
gefhfff wrote: post #5

Earlier quoted context omitted.

Is it easy now?

The technologies (Kafka, Kafka Connect, Kafka Streams, whatever you use) have stabilized somewhat but Event Sourced systems are as complex as ever.

Kafka and friends can be used in reactive pattern paradigm ( understood as here https://www.youtube.com/watch?v=eRxLfUIMJwk&t ) - only partially overlaping ES idea, which in general has much more sense to me.

Re: Event Sourcing Is Hard (2019)

#34

Post by EdwardDiego »

Most common issue I've ever encountered with event sourcing is that devs submit events that when replayed, lead to invalid states.

And if you're using an immutable log (Kafka/Pulsar/Liftbridge etc.) fixing those mistakes is hard.

And given you're basically recording interim states of a state machine, how do you verify that event Z being submitted after event Y is a valid state transition?

Validating a record against a schema is trivial. But validating three consecutive events against the state machine? I'm not aware of a good way to do that cheaply.

I've done it before using apps that sat in the data pipeline, and maintained state to reject invalid transitions, but that can end up being GiB of state, then you have to persist that across restarts or horizontal scaling.

It's all doable, but it costs. Is the event streaming paradigm actually worth the cost? I haven't been convinced yet.

But, I'm a big fan of streaming DB change events for auditing purposes - but those are coming from software that already enforces valid state changes.

So maybe the answer is - every event has to be submitted via Postgres.

Re: Event Sourcing Is Hard (2019)

#36

Post by Lhiw »

akiselev wrote: post #19

I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparin…

We distributed systems engineers have a saying, don't.

Re: Event Sourcing Is Hard (2019)

#37

Post by dm3 »

geocar wrote: post #20

I think one major problem is that "Event Sourcing" can mean subtly different things to different people. > The idea of a keeping a central log against which multiple services can subscribe and publish is insane. This really doesn't mean "Event Sourcing" to me, it sounds like enterprises that have decided Event Sourcing == Kafka (or some cloud-hosted IOT-branded variant) and treat the central broker/coordinators/confl…

> I think one major problem is that "Event Sourcing" can mean subtly different things to different people.

You're right. Here's a list of posts[1] about Event Streaming which claim to be about Event Sourcing. The discussed article is in the list.

[1]: https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...

Re: Event Sourcing Is Hard (2019)

#38

Post by Liron »

I think there's a middle option between (1) building your application up from a low-level stream-of-events data model and (2) having your application full of logic to snap from one high-level state to another without a good data-layer audit trail:

I'm talking about indexes.

Yes, the index part of your database is amazing, because it constantly snaps itself into various states, and you know exactly why: because it needs to maintain itself in the same deterministic relationship with your models.

I think that if databases would support more complex and composable indexes, like providing the ability to index join queries, then application programmers would be more nimble and bug-free working with their data.

I've written more about this here: https://lironshapira.medium.com/data-denormalization-is-brok...

Re: Event Sourcing Is Hard (2019)

#39

Post by geoduck14 »

bruth wrote: post #26

The title here is being overlooked, its not _terrible_, its _hard_. Also this quote from the article: > The bulk of these probably fall under "he obviously didn't understand X," or "you should never do Y!" in which case you would be absolutely right. The point of this is that I didn't understand the drawbacks or pain points until I'd gotten past the "toy" stage. Most of what they discussed was indeed not great, but t…

I've never heard of the term Event Sourcing. I AM familiar with Event Streaming. Some of what this article talks about surprised me - and I think it is because I don't fully understand the difference between Event Sourcing and Event Streaming.

Is this article a fine summary?: https://developer.confluent.io/learn-kafka/event-sourcing/ev...

Re: Event Sourcing Is Hard (2019)

#40

Post by Lhiw »

EdwardDiego wrote: post #34

Most common issue I've ever encountered with event sourcing is that devs submit events that when replayed, lead to invalid states. And if you're using an immutable log (Kafka/Pulsar/Liftbridge etc.) fixing those mistakes is hard. And given you're basically recording interim states of a state machine, how do you verify that event Z being submitted after event Y is a valid state transition? Validating a record against…

> Most common issue I've ever encountered with event sourcing is that devs submit events that when replayed, lead to invalid states.

This doesn't sound like event sourcing. It's likely your events aren't events or someone changed what your event handlers do at a later date instead of releasing a new version of the event.

Post reply on HN