Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a CQRS/ES web application in Elixir using Phoenix

10consulting.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Building a CQRS/ES web application in Elixir using Phoenix

#51

Post by davidcaseria »

ehsanu1 wrote: post #50
davidcaseria wrote: post #47

Earlier quoted context omitted.

I work for a finance startup that uses Akka Persistence to implement an event sourced architecture. It has worked pretty well for us, much better than the original CRUD system. Akka can support a variety of different databases via journal plugins which has allowed us to maintain using a SQL database instead of being forced to adopt an unproven database. The one area we've struggled with in the architecture is the que…

Just curious, what issues did you have that were solved by an independent projection application?

We're currently in the midst of implementing the independent application so we have not definitely solved our projection issues yet.

The main issue we have been running into is a performance bottle neck after restarting the application. Currently, when the application restarts (i.e. after a deployment) the projections may not update until several minutes. Our hypothesis is that if we can run projections independent of the command side of the application they can run indefinitely (i.e. as Spark jobs) since in theory a projection should not be updated. If we need to make changes to a projection then we will deploy another version to run side-by-side with the existing projection until the consumers migrate to the new version.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#52

Post by [deleted] »

eric_b wrote: post #36
jen20 wrote: post #32

Earlier quoted context omitted.

No shit... if you listen to the actual talks, that is EXACTLY what they recommend. Use the patterns where appropriate, don't use it where it's not appropriate. Also consider _where_ the advice came from - finance, gambling, healthcare and so forth. As for "not production ready" with regards to Event Store specifically, we have had zero reported incidents of data loss which were catastrophic failure of the hardware (o…

One of the projects I worked on was in finance. The fact that we were going to get "a free audit log" from the architecture made everyone so excited. After the CQRS/ES project failed (I was on cleanup duty) we used a more traditional arch. To handle the audit log we just had a separate table. ("customer" table had "customer_audit" table. Both were written to in transaction. Solved.)

[deleted]

Re: Building a CQRS/ES web application in Elixir using Phoenix

#53

Post by hcarvalhoalves »

eric_b wrote: post #35
jen20 wrote: post #33

Earlier quoted context omitted.

I would be interested to know why you consider ES "not production ready". There have been zero reported incidents of data loss in stable released versions which were not as a result of catastrophic failure of the hardware on which it was running (though most people's poor understanding of Azure has caused more problems than everything else put together). Furthermore, this should be no surprise given the testing proce…

We had no data loss incidents. My concern was around tooling and maintenance of the database. Once you had to start clearing out bad events in dev environments or doing common administrative chores there were many holes. I never did find out how to remove specific problem events. We ended up just purging the store every time something went wrong in lower environments. We had no solution for this problem in prod. How…

    How would you fix an event that should not have gotten in to the store?
- Reverse transaction (like in accounting)

- Replay events and filter/modify the problematic events into a new event store

    Fortunately the projects always failed well before we got in to any kind of production with real users.
That the project failed and the above was still an open question for you explains a lot.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#54

Post by slobdell »

btown wrote: post #49
slashdotdash wrote: post #37

Earlier quoted context omitted.

Pushing read model updates back to the client using a two way communication channel is one technical solution. I want to experiment with using Phoenix channels[1] to solve this. I think that has potential for easing the UI/UX concerns. You post a command from the web front-end and subscribe to receive updates for the read model you're looking at. Domain events can drive the client notification. [1] http://www.phoenix…

If you can write your read model into Mongo, then you can use Meteor to build a real time interface extremely quickly; it tails the database log and dispatches updated records to subscribers practically instantly over websockets, no need for the event processing code to know about how to map to frontend queries. We use this for our production CRM, albeit for internal users. No doubt Phoenix would be more performant a…

alas

Re: Building a CQRS/ES web application in Elixir using Phoenix

#55

Post by biot »

realntl1 wrote: post #38
eric_b wrote: post #34

Earlier quoted context omitted.

I was waiting for the "you're doing it wrong" guy to show up. You win! Yes, some of the systems used subscriptions too, which had their own set of issues. Additionally, domains are almost never completely understood. Even if they're well understood today, things will change tomorrow. CQRS/ES in your own words is not good when requirements change. Well guess what? That's every system I've ever worked on. If you've had…

You didn't succeed at building a CQRS/ES system despite several attempts. Why aren't you asking "what am I doing wrong?" instead of presuming that your personal experiences are sufficient to render informed judgement? > Additionally, domains are almost never completely understood. Even if they're well understood today, things will change tomorrow. CQRS/ES in your own words is not good when requirements change. Well g…

Maybe they've been asking since 2010 and, not having received a satisfactory solution from the experts in the field, have stripped all the projects of CQRS/ES and gone back to what works well. There comes a point in time where you stop asking and move on, and expecting them to re-ask on HN is a poor presumption on your part, leading to an uninformed judgment.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#56

Post by Haldir »

karmajunkie wrote: post #28
UK-AL wrote: post #25

Earlier quoted context omitted.

CQRS by itself does not imply using ddd or es.

Yeah, fair enough, but if you're not using ES then the names of messages don't matter a whole lot because you don't have to live with them forever. (Edit: ok, they matter some , in the way names of variables and apis matter.)

Dino Esposito describes an "historical" crud System in a series in msdn magazine https://msdn.microsoft.com/magazine/mt703431 This is basically ES with crud. Not saying ES with crud is the best example, but for data which requires Audit Trail logic it actually works fairly well.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#57

Post by modarts »

eric_b wrote: post #6

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I ha…

I'm curious to hear what experiences you had with Datomic that led you to the conclusion that it wasn't production ready.

Re: Building a CQRS/ES web application in Elixir using Phoenix

#58

Post by kiwidrew »

slashdotdash wrote: post #21
eric_b wrote: post #6

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn't make it work. There's more than one flavor of this particular arch, but Event Sourcing in general is simply not very useful for most projects. I'm sure there are use cases where it shines, but I ha…

I'm the author of this article. It sounds like you have some valuable, real-world experience with CQRS/ES. I'd love to read more about the difficulties you've faced, and overcome. For migration of immutable events, there's a good research paper[1] that outlines five strategies available: multiple versions; upcasting; lazy transformation; in-place transformation; copy and transformation. The last approach even allows…

Thanks for the link to the research paper, it's a good read. Are you aware of any event sourcing frameworks/datastores that use one (or more) of these strategies to tackle the problem of schema evolution?

Re: Building a CQRS/ES web application in Elixir using Phoenix

#59

Post by jgeraert »

lucisferre wrote: post #7
arcbyte wrote: post #2

> CQRS library You missed the whole point of CQRS

I know Greg personally, and I have heard him say things like "you don't need a library" in the past. However despite the fact that the basics are simple there are a lot of repeatable concepts that make sense to centralize around and even standardize. There is nothing wrong with a library, just as it is possible to do it without. One thing that absolutely needs library/drivers is persistence of the event store. I say…

> One thing that absolutely needs library/drivers is persistence of the event store. I say this having built my own as well as contributing to open source ones in the past. It is still a hard and not well solved problem to do this well. Again it can be simple, but in the real world it isn't usually that simple.

Would it also be such a hard thing to do if you can delegate the actual persistence to something like a rdbms? What are the typical pitfalls?

Re: Building a CQRS/ES web application in Elixir using Phoenix

#60

Post by wmfiv »

kiwidrew wrote: post #58
slashdotdash wrote: post #21

Earlier quoted context omitted.

I'm the author of this article. It sounds like you have some valuable, real-world experience with CQRS/ES. I'd love to read more about the difficulties you've faced, and overcome. For migration of immutable events, there's a good research paper[1] that outlines five strategies available: multiple versions; upcasting; lazy transformation; in-place transformation; copy and transformation. The last approach even allows…

Thanks for the link to the research paper, it's a good read. Are you aware of any event sourcing frameworks/datastores that use one (or more) of these strategies to tackle the problem of schema evolution?

Axon Framework is a very mature solution for ES on the JVM.

http://www.axonframework.org/

Here's the section of their manual related to this. http://www.axonframework.org/docs/2.4/repositories-and-event...

Akka Persistence is the other JVM ES framework that I'm familiar with. it supports upcasting as well.

http://doc.akka.io/docs/akka/2.4/java/persistence-schema-evo...

Post reply on HN