Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

McDonalds Event Driven Architecture

medium.com

Post reply on HN
Print view View original

91–100 of 134 posts

Re: McDonalds Event Driven Architecture

#91

Post by nick0garvey »

foolfoolz wrote: post #43

> Standby Event Store: To avoid loss of messages in the event the MSK is unavailable, the platform is wired with a standby data store, where it writes events onto a database . The architecture provides tools and utilities to read messages and publish them back onto MSK, once it’s available. on one hand aws msk is good enough for an enormous application like mcdonald’s. on the other they need a backup database just to…

Cloud services do go down and it's out of your control when they do. If something must work, you need redundancy.

Kafka is often used for financial applications that must not miss events, so having a backup buffer is a reasonable strategy for those use cases. Things like tracking data is likely not worth backing up due to high data volume and low external visibility when data is dropped.

Re: McDonalds Event Driven Architecture

#92

Post by strokirk »

b20000 wrote: post #27
ramenmeal wrote: post #20

Earlier quoted context omitted.

In your monolith if you offload a request to a queue and have a background process processing the queue messages, what do you call that module processing messages? Is it a component of your monolith? I think a lot of teams would call it a microservice, I could see some people considering it a component of a service. Unless you're confounding microservices w/ async architectures and saying to drop asynchronous pattern…

I don't need another process. I can use a thread. If the teams you are talking about never heard of threads and only know about microservices, then there is something seriously wrong with their CS education. Maybe they all were hired via leetcode. That could explain it. I'm not confounding anything. Distributed programming has its applications and uses, but if you don't have a good reason to use it, then don't, and u…

My CS education (Stockholm, KTH) didn’t include anything about web service architecture, and information about threads was about how they are implemented in the OS at a low level, not how to use them effectively. I think this stuff is normally picked up after working in the industry.

Re: McDonalds Event Driven Architecture

#93

Post by BoorishBears »

andy800 wrote: post #69
BoorishBears wrote: post #63

Earlier quoted context omitted.

I mean you wrote all this justification, but 30k per second is still practically nothing compared to the complexity described in the article? Taking something like Postgres and sprinkling in some strategic use of Redis would handle their usecase with horizontal scaling pretty reliably... What it wouldn't do is let you add Kafka to your resume.

To be clear, I was not justifying the current architecture, I specifically wrote "an entirely different architecture could very easily work much better." I was pointing out, however, that, as is often the case, the initial estimates in a typical "why do they need all this stuff" post, likely underestimated the transaction volume by possibly 10x. Perhaps 3000 or 30000 transactions per second could run on the same syst…

I mean if you got nitpicky 3000 or 30000 transactions doesn't tell you anything... but in this kind of evaluation you need to think dimensionally. That's why I intentionally assumed all of their traffic shows up in one contiguous block of 8 hrs across all locations every day: that added a massive fudge factor even bigger than the number you're focusing on...

> But I doubt you'd find any Fortune 100 company relying solely on Postgres and Redis.

I mean, yeah?

Across every system they use of course that wouldn't be it: what would be generating the data that goes into them? Where would the data that goes in be going out?

I'm simply referring to their "glue" for day to day operations, which here is a pubsub system built on Kafka. Most organizations of a certain size start to pick up some set of technology that new efforts default to being built on top of if only to have access to what everyone else is doing... that's essentially what AWS started off as before it was spun out from internal usage

-

But more importantly, Fortune 100 is a very random pairing of problem spaces. I mean you won't find any built solely on Postgres and Redis for the very obvious reason I mentioned above... but you will find billions of dollars in revenue on even more boring stuff than that. The number of Oracle shops using repackaged technology that makes Postgres look like Cloud Spanner is staggering.

I find the opposite of what you do, that people tend to overestimate what it takes to handle large amounts of data reliably. And I think it's because you need some experience with this stuff to understand why you can't just think in terms of "underestimated the transaction volume by possibly 10x"(hint: 10x can mean 3 million => 30 million).

What happens is people hear that system A is going to need to go from 3,000 to 30,000, then start to architecture the way someone going from 3 million to 30 million should have, and suddenly you're building out a system that's less reliable, more expensive, and just generally worse except for what shows up on resumes.

Re: McDonalds Event Driven Architecture

#95

Post by cgio »

hamandcheese wrote: post #78
foolfoolz wrote: post #43

> Standby Event Store: To avoid loss of messages in the event the MSK is unavailable, the platform is wired with a standby data store, where it writes events onto a database . The architecture provides tools and utilities to read messages and publish them back onto MSK, once it’s available. on one hand aws msk is good enough for an enormous application like mcdonald’s. on the other they need a backup database just to…

We use kinesis a lot at work, and some services are architected to write events to a Postgres table which eventually gets dumped to kinesis, while other services write directly to kinesis. Guess which services fared better during the last kinesis outage?

What's the point of using kinesis in this case? Why can you not consume from postgres. My understanding would be that you use kinesis for impedance matching on writes. If you can already write everything on Postgres reliably I cannot see any immediate use case for Kinesis. I could see the other way around, i.e. having kinesis as a resilience backup for postgres being a more valid concern.

Re: McDonalds Event Driven Architecture

#96

Post by geysersam »

rbosinger wrote: post #22

Both my partner and I always have trouble with the McDonald's app (for ordering). I use Android, she uses iOS. As a developer, I've said to myself "This feels like a React Native app that's calling into a mess of microservices" (having worked on that type of project more than once myself). Anyway, I only skimmed the article, but I had a chuckle seeing the title of this article pop up on HN at all.

I wished more people used a regular website for these kinds of applications.

It's a form. What's the problem?

Re: McDonalds Event Driven Architecture

#97

Post by joezydeco »

filmgirlcw wrote: post #84
bitshiftfaced wrote: post #58

Earlier quoted context omitted.

I would never had guessed that Chic-fil-A was considered a contender to McDonald's in technical competency. I drive up and a person standing outside takes my order and asks my name. Then I drive up some more and another person asks me my name and punches something into a tablet, and then I drive up some more and get to the window. This is 2022. Businesses already have queueing software that automatically takes images…

They have humans do it because it is faster. Significantly faster, in fact, to go car to car, rather than have each car drive up to a window to order. It also allows them to have multiple lines running at once. And it is probably just as fast to have to person handing you food just ask your name so they can make sure to pass it off efficiently. Chick-fil-A is the only chain where you will reliably see a massive line…

CFA has found lots of small incremental improvements that speed up the drive thru process significantly.

MCD did the double drive thru ordering first, but my local CFA now has two pickup lines as well, with a heated covered roof. The window has been replaced with a wide open door and employees walk the orders to each car. Simple and it works great.

Re: McDonalds Event Driven Architecture

#98

Post by joezydeco »

filmgirlcw wrote: post #89
mbg721 wrote: post #45

Earlier quoted context omitted.

Thinking about it, that makes sense, given how gargantuan Chick-Fil-A is, but the two chains leave very different impressions to me. Chick-Fil-A feels like they're going out of their way to make it seem like there are friendly humans running everything, to the point of having employees with tablets stand outside at the drive-thru to take your order; lately, they've even been laying off the robotic "my pleasure" thing…

For better or worse, Chick-fil-A is very obsessive about its standards and its quality. I grew up in Atlanta (where CFA was founded/based) and have had lots of friends who have worked for CFA over the years in various positions. Even in high school, CFA was considered the best fast-food job you could get (and better than some retail jobs) because it paid better than almost anyone else. The downside was there was a st…

It's my understanding that CFA franchisees are only allowed to own one store (unlike MCD), and must work in their restaurant full-time.

That's a significant difference from how MCD stores are operated and it shows.

Re: McDonalds Event Driven Architecture

#99

Post by tom_walters »

Did AWS just pay them to write this? I was hoping for an interesting article exploring the complexities of a massively distributed and high-throughout system, but I just read “we connected these managed AWS services together and it’s cool”

Hopefully further instalments might actually talk about the problems they faced building this out, and their unique challenges.

Post reply on HN