Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing (2017)

arkwright.github.io

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Event Sourcing (2017)

#61

Post by zbentley »

_tkzm wrote: post #42

99,99% new people that come to event sourcing and 9/10 of those who have been in the event sorucing already, make the huge mystake of taking event store concepts from other people that took it from other people and in the end that is where everyone fails. even big names like greg and his praised "eventstore" project. if you are new to ES, great, you have no baggage. do not read any technicalities about the event stor…

There are a lot of assertions there about poor quality, "everyone" failing, and concepts being completely flawed.

Could you explain the rationale for those assertions, and expand on why rolling your own avoids those pitfalls?

Re: Event Sourcing (2017)

#62

Post by randolorian »

pookeh wrote: post #17
imglorp wrote: post #11

Earlier quoted context omitted.

Oh, why? Holding one key for every human on earth would fit in 8 TB plus read-mostly replicas. You'd delete their key if they made GDPR removal request and the keyless, encrypted, immutable entries would be entombed in place.

What do you do when the data involves multiple people?

Why would it? Typically you would have one stream per user.

Re: Event Sourcing (2017)

#63

Post by bsaul »

I don't understand how the uniq service would be able to scale horizontally.

How would you load balance calls to uniq to different servers and make sure there's still coordination to ensure unicity of values ?

Either you keep relying on the logs for replica syncing, but then the service can't answer in a synchronous manners. Or you need some synchronous distributed lock ? But then you still have the problem associated to locking described previously in the same article.

Re: Event Sourcing (2017)

#64

Post by joostdevries »

lukev wrote: post #20

A word of caution for anyone considering an event-sourced architecture. I was on a large government project where the decision was made to use event sourcing, and it was disastrous. It ended up being a big contributor to several years of time and cost overruns. The reason is that for event sourcing to work, you have to have a pretty good idea of your application's requirements up front. It's simply not conducive to a…

Your word of caution is important. For some projects it's not initially clear what the right data model and bounded contexts are. And for some projects it's just overhead. But the converse is also true: for some projects/systems it turns out that it being events first is the _only_ way for it to work. I've encountered that the last few years with systems in logistics that show an integral view across parties.

The way we approached that is that we started with a standard system and refactored to event processing approach when the decomposition into bounded contexts was clear.

So in that sense it's similar to the right way of approaching microservices: start out with a 'monolith' that you split up.

Re: Event Sourcing (2017)

#65

Post by the_gipsy »

rodocite wrote: post #4
xtagon wrote: post #2

Great article. One thing that wasn't pointed out that might be of interest to someone learning about Event Sourcing is that it introduces some challenges if you are to be compliant with GDPR and similar laws. For example, if your event log is immutable, and you use it as an audit log, then by nature you are not ever deleting data. There are solutions to this (for example, crypto-erasure), but it can be non-trivial to…

I'm pretty sure you can "soft-delete" for GDPR compliance. So this concern is sort of a non-issue. Besides that: 1. If you're using an immutable structure, as long as you use references, you can obfuscate data. Blockchains ran into this problem before GDPR requirements and that's essentially all they do. 2. ^ "Update" strategy for event sourcing is the same as above. Essentially a copy of the log or the log slice the…

"soft-delete" is not GDPR compliant.

Re: Event Sourcing (2017)

#66

Post by dgregd »

m45t3r wrote: post #52
GordonS wrote: post #50

Earlier quoted context omitted.

This sounds like working around event sourcing - what value does event sourcing add here, vs simply not using it at all for user accounts?

For this specific case, nothing, however let's say you need to know when the user got a specific scope and from who. You probably can answer this question in SQL if you prepared your database to do it (i.e: a changeset table), however in an event sourcing architecture you would gain this information for free. However, just because you're using event sourcing doesn't mean you don't have a database with the current sta…

In Ruby on Rails one can add just one line of code to achieve the same result.

  class User
That solution is good enough for most use cases and doesn't require highly skilled architects and developers to implement Event Sourcing properly.

Re: Event Sourcing (2017)

#67

Post by randolorian »

seer wrote: post #57
randolorian wrote: post #54

Earlier quoted context omitted.

Best practices around this have already been established. Most if not all event stores - which Kafka is not - have a concept called 'position.' You save the position atomically along with whatever you did with the message. Then if you crash, you simply ask for all messages starting from that position. If you have a new service (or a new projection), your position is 0 so you get everything.

That is indeed the case, and this is what Kafka calls unlimited retention topics. This however hampers schema evolution significantly as you're not allowed to make backwards incompatible changes. Or rather if you allow those changes, every service would need to be able to handle every schema throughout time. If you make the retention to only several days this would mean you can evolve your schema with even breaking c…

I feel like people are running into these problems because they want to pretend that a message broker is an event store. I could try to shovel a star schema into MongoDB too, but why would I want to?

Keeping data only in the latest schema is dangerous. We have no idea what data the business will find useful years down the line. By only having whatever is in the latest schema, you may have thrown valuable data away.

Re: Event Sourcing (2017)

#68

Post by tunesmith »

randolorian wrote: post #62
pookeh wrote: post #17

Earlier quoted context omitted.

What do you do when the data involves multiple people?

Why would it? Typically you would have one stream per user.

Event: "User A sent Message X to User B". (I think the answer here is again to have PII be in other repositories, and only refer to them by id in the log.)

Re: Event Sourcing (2017)

#69

Post by specialist »

tunesmith wrote: post #3
xtagon wrote: post #2

Great article. One thing that wasn't pointed out that might be of interest to someone learning about Event Sourcing is that it introduces some challenges if you are to be compliant with GDPR and similar laws. For example, if your event log is immutable, and you use it as an audit log, then by nature you are not ever deleting data. There are solutions to this (for example, crypto-erasure), but it can be non-trivial to…

I saw the Akka people talking about this on twitter once, I think they were theorizing that encrypting the data in the log would be sufficient, because then "deleting the key" could be interpreted as the deletion of the record (even though the useless data still exists in the log). But I'm not sure this was ever legally validated?

Just like password stores. Store salted and hashed password value, vs actual password.

Cleverly applying this strategy to all sensitive datas is the Translucent Database thesis.

Re: Event Sourcing (2017)

#70

Post by specialist »

pookeh wrote: post #17
imglorp wrote: post #11

Earlier quoted context omitted.

Oh, why? Holding one key for every human on earth would fit in 8 TB plus read-mostly replicas. You'd delete their key if they made GDPR removal request and the keyless, encrypted, immutable entries would be entombed in place.

What do you do when the data involves multiple people?

Good question. Create alias proxies?
Post reply on HN