71–80 of 166 posts
The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.
I agree with the author completely. I worked on a fairly large system using event sourcing, it was a never-ending nightmare. Maybe with better tooling someday it will be usable, but not now. Events are pretty much a database commit log. This is extremely space innefficient to keep around. And not nearly use useful as you might think. Re-runs need to happen pretty often as you change how events are handled. Even in ou…
Aren't you supposed to take Snapshots every now and then to solve that problem?
One of the problems I think I see with event sourcing is its inability to scale. You have to guarantee the order of events, right? How do you do that in a large scale distributed system with eventual consistency, without incurring an insane synchronization time penalty? I'd genuinely love to hear if you have a good solution for this, because if you do I have a use case I need it for, so this ain't a troll comment!
For the former, you might use something like [ZooKeeper](https://www.elastic.co/blog/found-zookeeper-king-of-coordina...) in your consumers to control parallelisation.
For the latter, you order by time of insertion. I think you might have some contraints you haven't mentioned that may be the real problem you are trying to solve. What is distributed?
Datomic is, at its core, an event sourced datastore, and it works really well. I don’t think event sourcing is something that should be solved in application space — you wouldn’t write a database from scratch for your products yet implementing an in-house event sourcing system is for some reason more acceptable. All of these stories of failure are stories of teams bandaiding event sourcing on top of other databases n…
Datomic may be something like 'event sourcing' internally, but it works so well because it abstracts and hides the naked overreach that the article pounds against.
Datomic users think of it as a DB, not a stream of events. So I don't think that Datomic users are following the 'event sourcing' model.
I agree with the author completely. I worked on a fairly large system using event sourcing, it was a never-ending nightmare. Maybe with better tooling someday it will be usable, but not now. Events are pretty much a database commit log. This is extremely space innefficient to keep around. And not nearly use useful as you might think. Re-runs need to happen pretty often as you change how events are handled. Even in ou…
>Re-runs need to happen pretty often as you change how events are handled. Aren't you supposed to take Snapshots every now and then to solve that problem?
But event sourcing is not for every application. Event sourcing solves some, otherwise hard, problems at the cost of added complexity. You need to judge whether it pays off.
Event sourcing assumes particular size of an application. Too small application will pay a lot in complexity with no added benefits because the problem would otherwise be easily solvable without having to use event sourcing. Too large an application will pay a lot in complexity because your write path will be complex due throughput requirement.
Event sourcing requires dedication. You can't go half-way, mix event sourcing here with direct inserts there, for example. This is going to be hell of a complex environment to live in with worst of the two worlds. Event sourcing only solves problems if you use it 100%.
I don’t think what the author did sounds like event sourcing, as I think of it. His setup sounds more like pubsub. In all honesty I’m probably the one doing it wrong though.
My event stream isn’t typically consumed or listened to by arbitrary listeners. (Although it can be, in some rare cases) Each event is namespaced to a specific module. When you load a log, you provide singletons for each module. Each message is only received by one singleton. If you want to broadcast that message to another module you have to do that from the parent module.
Everything is totally imperative so there’s no ambiguity about what causes what.
If I need data on a different machine, or in the browser, I just send the log down as a procedure and run it with fresh singletons on the client.
I also don’t have one mega stream, I only mix modules that actually interact. If I have two pieces of data that aren’t connected, they each have their own log.
I don’t know, maybe I’m the one who’s not really doing event sourcing.
Like I said, I’m still somewhat at the toy stage, so maybe I will get to where OP is eventually. I do plan on doing log rewriting for compression. That might lead to some pain.
I also haven’t yet had to deal with sharding. I suspect there’s pain there.
My plan is to use the one-singleton/one-module per message namespace rule to keep things from getting too complex. I figure if the same module that consumes messages is responsible for rewriting them, maybe it won’t be too weird. We’ll see. I have a lot more prototyping to do.
If anyone is interested, the core module is “a-wild-universe-appeared” on NPM. It’s still in the 0.x.0 series so it could break. But the basic API is pretty simple and hasn’t changed much in a few months even though I’ve been using it regularly.
https://www.npmjs.com/package/a-wild-universe-appeared
I agree with the OP that it’s perhaps too soon to throw this kind of thing at production problems. We need more research on how stores like this integrate with other layers.