Ask HN: Has anyone fully embraced an event-driven architecture? • Hacker News Board

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#131

Post by Fiahil » Tue, Aug 03, 2021, 2:20 PM UTC

I work for a big AI consultancy. Most of the time we build ETLs for the data-engineering side, in a client driven capacity-building effort. We do this because our focus is on Data Science, not data engineering, and we often work in situations where the client doesn't have an existing data science platform. It's simpler to build, to handover and later to maintain.

In projects where the client already have a mature engineering and data science department, we bring the big guns! The scope is usually much larger, with several workstreams and involve production-ready deployments. In this situation we might build upon what the client already have (ETLs), or initiate a full event-driven transformation with a "backbone" team responsible for creating a platform, and several use cases building upon it. In the usual scenario, a team would want to start large computations or simulations upon recieving a trigger event from a monitoring system (model drift) or a human operator ("what would be the impact in € of a small decrease in parameter X over the next 7 days of forecasted sales"?).

Even-driven systems are much more robust than traditional ETLs with a central data warehouse, but they are also much more complex to understand and operate. In the end, we rarely deploy them because they cost us way too much engineering time compared to the benefits. That's mostly because we spend >70% of our time dealing with "security teams" and "access issues". Seriously.

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#132

Post by jimmaswell » Tue, Aug 03, 2021, 2:52 PM UTC

PinguTS wrote: post #61

fxtentacle wrote: post #53

Earlier quoted context omitted.

I fully agree with this, also that's still quite common in the embedded world. The user presses a button which sets a hardware event flag. CPU wakes up from sleep, checks all event flags, handles them, clears the interrupt bits and goes back to sleep. But using events like this requires a very tight integration between event producer and consumer, so I don't think this will translate well to distributed systems or mi…

There is an example where highly distributed, event-driven systems are used. Everyday we are using them. It the cars. In a car there are many distributed ECUs(1) that communicate in an event driven system. It was tried to use cyclic communication. But all those attempts failed in the long run. Because cyclic communication would required that all the independent ECUs are synced to each other, which is a very hard prob…

Electronics can be more trouble than they're worth in cars. If they made modern cars with no electronics in the engine or controls, maybe I'd buy new cars. It's frustrating when my old Cherokee won't start because the antitheft system is having some kind of conniption for example, and I have to go through this ritual of locking and unlocking the doors, reconnecting the battery with the key in the ignition, flipping this mystery switch the previous owner has no clue about but seems to contribute somehow. There's a process to disable this system by buying a new ECU or something but I haven't gotten around to it.

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#133

Post by kieranmaine » Tue, Aug 03, 2021, 2:57 PM UTC

sumtechguy wrote: post #127

kieranmaine wrote: post #52

Earlier quoted context omitted.

When you say you used a central schema registry, did you have a single repo containing all topics and schemas?

kafka has a 'schema registry' service. Typically they use avro/json to define the schema of each message. Think they added a couple of new types in recent versions. When you define your consumer/producer you also tell avro what the schema registry service URL is. Basically it helps you keep your messages in spec. It also has the ability for versioning so you can have different versions of messages in flight on the sa…

Thanks for the reply. We're using a schema registry but we generally have our schemas spread across multiple repos depending on who owns the topic. I was wondering if they centralized all their schemas under a single repo. I'd like to do this, but I wanted to get some other opinions on the subject.

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#134

Post by _3u10 » Tue, Aug 03, 2021, 3:11 PM UTC

Yes, most MVC platforms are event driven.

Since you mentioned schema breakage what I imagine you’re doing is inner platform effect as your API / database already supports everything you are trying to reinvent, just use Postgrest and views.

When you break a view you know you’re making incompatible changes. Stop right there and either version the view, or figure out how to add your feature without breaking the view.

It’s pretty easy to avoid making breaking API changes.

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#135

Post by sumtechguy » Tue, Aug 03, 2021, 3:31 PM UTC

kieranmaine wrote: post #133

sumtechguy wrote: post #127

Earlier quoted context omitted.

kafka has a 'schema registry' service. Typically they use avro/json to define the schema of each message. Think they added a couple of new types in recent versions. When you define your consumer/producer you also tell avro what the schema registry service URL is. Basically it helps you keep your messages in spec. It also has the ability for versioning so you can have different versions of messages in flight on the sa…

Thanks for the reply. We're using a schema registry but we generally have our schemas spread across multiple repos depending on who owns the topic. I was wondering if they centralized all their schemas under a single repo. I'd like to do this, but I wanted to get some other opinions on the subject.

I have seen several approaches.

1) the producer repo owns it. This has the benefit of only the producer makes those things and it is side by side. Downside if you have more than one producer repo with the same schema. When the producer code starts up it sets up the schema.

2) central repo. This is nice however tends to make it if you are using a micro service style more of pain to deploy as they all sorta need to move together. But works very nicely with mono repos. You can make it work with micro but it takes a bit more thinking. This turns your deploy into a two step process 'schema first' then code. That can work but again 'more thinking'.

3) consumer owned. The consumer basically says 'i will only grab these anything else I will consider error'. Works ok if you have 1 producer group and one consumer group and is effectively #1. But with several consumers it becomes a 'cut and paste job'. Or something may get out of sync, etc. When the consumer starts up it pushes the schema.

What I found best to get someone to decide is to figure out what is your version upgrade cycle. Is it one thing at a time? If it is 'everything goes every time'? Who 'owns the schemas?' So different styles will match your deployment process more than anything.

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#136

Post by kieranmaine » Tue, Aug 03, 2021, 3:43 PM UTC

sumtechguy wrote: post #135

kieranmaine wrote: post #133

Earlier quoted context omitted.

Thanks for the reply. We're using a schema registry but we generally have our schemas spread across multiple repos depending on who owns the topic. I was wondering if they centralized all their schemas under a single repo. I'd like to do this, but I wanted to get some other opinions on the subject.

I have seen several approaches. 1) the producer repo owns it. This has the benefit of only the producer makes those things and it is side by side. Downside if you have more than one producer repo with the same schema. When the producer code starts up it sets up the schema. 2) central repo. This is nice however tends to make it if you are using a micro service style more of pain to deploy as they all sorta need to mov…

Thanks for the feedback!

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#137

Post by resters » Tue, Aug 03, 2021, 3:44 PM UTC

There are many examples of a "hello world" for event-driven architecture, but there doesn't seem to be the equivalent of the "sinatra/express" of event-driven architecture, a minimalistic foundation to build a customized platform using the approach. Things like schema migration layer would nicely onto something like this.

Most event-driven systems are big company projects with a lot of legacy requirements and integration complexity, or they are narrowly tailored and hard to generalize.

I think that a simple template event-driven system that includes a small number of libraries and does something simple but interesting would be a big help.

I've been wanting to create an open source event-driven e-commerce system but haven't had the time.

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#138

Post by navd » Tue, Aug 03, 2021, 3:48 PM UTC

stonemetal12 wrote: post #121

lloydatkinson wrote: post #84

Earlier quoted context omitted.

I'm not sure you've understood EDA if you're suggesting that good logging is an alternative.

It isn't an alternative, it is a necessary part of understanding the system. You get an event and you have no idea where it came from. That is great from a composability stand point, anything can become an event emitter. On the other hand if that event comes in with bad data, you have no way to correct the issue because you have no idea where the event came from. The more event sources become disassociated from the e…

Thank you for saying the things I didn't think I needed to elaborate on :-)

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#139

Post by orekrogue » Tue, Aug 03, 2021, 4:13 PM UTC

EamonnMR wrote: post #104

orekrogue wrote: post #83

Earlier quoted context omitted.

What do you use to encode the messages? Avro, Proto or maybe just JSON? Most of the books and articles encourage the use of Avro, but I feel like the support in python isn't nearly as good as in other languages. Almost every solution operates on dictionaries instead of classes with annotations which doesn't seem like something you would want to use for a Py project in 2021...

We use Json in our Python projects and Avro in our Java projects. We have considered Avro, CSV, Protobuff or CapnProto for the Python projects but it's never been enough of a win for us to prioritize it. We switched to Pydantic from dictionaries.

Thanks for response! That's basically what i ended up with! Avro whenever possible and for python stick with pydantic. Except that at the same time i want to keep avro schema for those pydantic models and just convert them with fastavro, because then i dont have to rely on the quality of code/schema generators. A little sceptic about that though, as keeping the same schemas in 2 different technologies compatible and double (de)serialization might be troublesome.

Re: Ask HN: Has anyone fully embraced an event-driven architecture?

#140

Post by relaxing » Tue, Aug 03, 2021, 4:31 PM UTC

narag wrote: post #114

jfoutz wrote: post #41

You're not going to like the answer, but I think it captures some of what you're getting at. Windows 95. Old style gui programming meant sitting in a loop, waiting for the next event, then handling it. You type a letter, there's a case switch, and the next character is rendered on the screen. Being able to copy a file and type at the same time was a big deal. You'd experience the dead letter queue when you moved a wi…

Windows 95. Old style gui programming meant sitting in a loop, waiting for the next event, then handling it. The GUI worked in a single thread, but your whole program didn't need to do that. Being able to copy a file and type at the same time was a big deal. That's not correct. You just needed to create a separate thread for the file operation. For some programmers that was a big deal indeed, the same could be said f…

OP is talking about what you could do using the built in Windows utilities, not what is possible if you write your own file copy-and-note taking utility (which some people did, like Borland Sidekick.)