Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a CQRS/ES web application in Elixir using Phoenix

10consulting.com

Post reply on HN
Print view View original

31–40 of 81 posts

Re: Building a CQRS/ES web application in Elixir using Phoenix

#31

Post by slashdotdash »

karmajunkie wrote: post #26
slashdotdash wrote: post #21

Earlier quoted context omitted.

I'm the author of this article. It sounds like you have some valuable, real-world experience with CQRS/ES. I'd love to read more about the difficulties you've faced, and overcome. For migration of immutable events, there's a good research paper[1] that outlines five strategies available: multiple versions; upcasting; lazy transformation; in-place transformation; copy and transformation. The last approach even allows…

Good link! Looking forward to reading that. I've been following your projects on Github for awhile, good work—I don't necessarily agree with all of the design choices but we've built on the eventstore at work and I'm going to be using it on another project in the near future.

Please do feedback your ideas to improve these open-source projects. I'd be interested to find out how you're using CQRS/ES in Elixir.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#32

Post by jen20 »

jbogard wrote: post #23
eric_b wrote: post #6

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I ha…

Wow, this succinctly sums up our experience. Fun to develop against, absolute nightmare to support in production. The only place I'd recommend it these days are where the business views their state as an event stream, maybe finance/stocks. Not developer-forced-events like "customer address updated" or "user email changed". Even workflow systems I've dealt with, the business doesn't view their state as an event stream…

No shit... if you listen to the actual talks, that is EXACTLY what they recommend. Use the patterns where appropriate, don't use it where it's not appropriate. Also consider _where_ the advice came from - finance, gambling, healthcare and so forth.

As for "not production ready" with regards to Event Store specifically, we have had zero reported incidents of data loss which were catastrophic failure of the hardware (or, usually the cloud) it was running on.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#33

Post by jen20 »

eric_b wrote: post #6

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I ha…

I would be interested to know why you consider ES "not production ready".

There have been zero reported incidents of data loss in stable released versions which were not as a result of catastrophic failure of the hardware on which it was running (though most people's poor understanding of Azure has caused more problems than everything else put together). Furthermore, this should be no surprise given the testing process[1] continually run.

[1]: https://geteventstore.com/blog/20130708/testing-event-store-...

Re: Building a CQRS/ES web application in Elixir using Phoenix

#34

Post by eric_b »

karmajunkie wrote: post #19
eric_b wrote: post #12

Earlier quoted context omitted.

I detailed some of the problems I saw in another thread, but one other thing that gets challenging is the CQRS side (if you choose to keep everything async). For example, say you fire an "address change" event. That gets sent to the event store for eventual projection in to a medium you can actually query from (realtime querying of the event store itself is the road to very bad places, I promise). So now your event i…

> Of course, this can be made to work, but once you start talking edge cases and the need to support standard UX paradigms, polling for every update and handling error scenarios in this way becomes painful. This really makes me think you—and the originators of these projects you're lambasting—are working from an incomplete understanding of how to apply CQRS and ES. If you're applying CQRS in a fully async fashion, po…

I was waiting for the "you're doing it wrong" guy to show up. You win!

Yes, some of the systems used subscriptions too, which had their own set of issues.

Additionally, domains are almost never completely understood. Even if they're well understood today, things will change tomorrow. CQRS/ES in your own words is not good when requirements change. Well guess what? That's every system I've ever worked on.

If you've had success building non-trivial CQRS/ES applications I'd love to hear more specifics about how you solved all the other issues I've presented.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#35

Post by eric_b »

jen20 wrote: post #33
eric_b wrote: post #6

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I ha…

I would be interested to know why you consider ES "not production ready". There have been zero reported incidents of data loss in stable released versions which were not as a result of catastrophic failure of the hardware on which it was running (though most people's poor understanding of Azure has caused more problems than everything else put together). Furthermore, this should be no surprise given the testing proce…

We had no data loss incidents. My concern was around tooling and maintenance of the database. Once you had to start clearing out bad events in dev environments or doing common administrative chores there were many holes. I never did find out how to remove specific problem events. We ended up just purging the store every time something went wrong in lower environments.

We had no solution for this problem in prod. How would you fix an event that should not have gotten in to the store? (Wrong contract, bad data etc) I understand it shouldn't happen, but in the real world all kinds of things go wrong.

I can't imagine running a massive system on something like event store. Fortunately the projects always failed well before we got in to any kind of production with real users.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#36

Post by eric_b »

jen20 wrote: post #32
jbogard wrote: post #23

Earlier quoted context omitted.

Wow, this succinctly sums up our experience. Fun to develop against, absolute nightmare to support in production. The only place I'd recommend it these days are where the business views their state as an event stream, maybe finance/stocks. Not developer-forced-events like "customer address updated" or "user email changed". Even workflow systems I've dealt with, the business doesn't view their state as an event stream…

No shit... if you listen to the actual talks, that is EXACTLY what they recommend. Use the patterns where appropriate, don't use it where it's not appropriate. Also consider _where_ the advice came from - finance, gambling, healthcare and so forth. As for "not production ready" with regards to Event Store specifically, we have had zero reported incidents of data loss which were catastrophic failure of the hardware (o…

One of the projects I worked on was in finance. The fact that we were going to get "a free audit log" from the architecture made everyone so excited.

After the CQRS/ES project failed (I was on cleanup duty) we used a more traditional arch. To handle the audit log we just had a separate table. ("customer" table had "customer_audit" table. Both were written to in transaction. Solved.)

Re: Building a CQRS/ES web application in Elixir using Phoenix

#37

Post by slashdotdash »

eric_b wrote: post #12
baconforce wrote: post #10

Earlier quoted context omitted.

Would love to hear about your experience and why CQRS/ES failed those projects, and what type of technical complexities it introduced.

I detailed some of the problems I saw in another thread, but one other thing that gets challenging is the CQRS side (if you choose to keep everything async). For example, say you fire an "address change" event. That gets sent to the event store for eventual projection in to a medium you can actually query from (realtime querying of the event store itself is the road to very bad places, I promise). So now your event i…

Pushing read model updates back to the client using a two way communication channel is one technical solution. I want to experiment with using Phoenix channels[1] to solve this. I think that has potential for easing the UI/UX concerns. You post a command from the web front-end and subscribe to receive updates for the read model you're looking at. Domain events can drive the client notification.

[1] http://www.phoenixframework.org/docs/channels

Re: Building a CQRS/ES web application in Elixir using Phoenix

#38

Post by realntl1 »

eric_b wrote: post #34
karmajunkie wrote: post #19

Earlier quoted context omitted.

> Of course, this can be made to work, but once you start talking edge cases and the need to support standard UX paradigms, polling for every update and handling error scenarios in this way becomes painful. This really makes me think you—and the originators of these projects you're lambasting—are working from an incomplete understanding of how to apply CQRS and ES. If you're applying CQRS in a fully async fashion, po…

I was waiting for the "you're doing it wrong" guy to show up. You win! Yes, some of the systems used subscriptions too, which had their own set of issues. Additionally, domains are almost never completely understood. Even if they're well understood today, things will change tomorrow. CQRS/ES in your own words is not good when requirements change. Well guess what? That's every system I've ever worked on. If you've had…

You didn't succeed at building a CQRS/ES system despite several attempts. Why aren't you asking "what am I doing wrong?" instead of presuming that your personal experiences are sufficient to render informed judgement?

> Additionally, domains are almost never completely understood. Even if they're well understood today, things will change tomorrow. CQRS/ES in your own words is not good when requirements change. Well guess what? That's every system I've ever worked on.

It's still possible to design individual components that don't require constant churn of their application state. Most software teams are incapable of this, and event sourcing is not for them, even in domains where it shines (like finance).

In my experience, when teams have solid leadership, you can get your software pretty close to the target the first time you build it. Minor course corrections are straightforward, if sometimes tedious. When the business experiences big pivots, much of what you've built can be reused providing it's modular and does not make assumptions about the overall system. The rest can be discarded.

That's a big departure from the topic at hand, but my point is that if your software isn't modular, event sourcing in particular will amplify the pain you feel.

I recommend that anyone thinking about doing CQRS/ES find someone who is an expert to help guide them or their team.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#39

Post by eric_b »

UK-AL wrote: post #20
eric_b wrote: post #6

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I ha…

I've worked on a successful CQRS project. It didn't use es though. Probably one the nicest systems I've ever helped build. We had a very good lead though.

I like thinking about applications in terms of read-side and write-side. I'm not super enamored with async CQRS, but I am a huge fan of APIs and data models where the developers realized that how I consume the data is wildly different than how I update it.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#40

Post by slashdotdash »

lucisferre wrote: post #7
arcbyte wrote: post #2

> CQRS library You missed the whole point of CQRS

I know Greg personally, and I have heard him say things like "you don't need a library" in the past. However despite the fact that the basics are simple there are a lot of repeatable concepts that make sense to centralize around and even standardize. There is nothing wrong with a library, just as it is possible to do it without. One thing that absolutely needs library/drivers is persistence of the event store. I say…

> If CQRS/ES wants to make it into mainstream it needs to think about tooling.

I agree with this entirely. CQRS/ES applications really help from tailored tooling. Tracking chains of events using correlation and causation identifiers. Auditing commands. Projecting events out to datastores like Elasticsearch. Then using Kibana[1] to create dashboards to drill down into the data. I'd like to go into this in more detail in the future. Revisit this case study after 6-12 months and describe the pitfalls, and countermeasures. Identify tools that allow you to monitor the system proactively.

[1] https://www.elastic.co/products/kibana

Post reply on HN