Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions of Messages a Day – Yelp's Real-Time Data Pipeline

engineeringblog.yelp.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#31

Post by eellpp »

z3t4 wrote: post #8

100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance. Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up in…

Applying the 80/20 principle: 80% of traffic occurs in the 20% of years time. Which is around 12 reviews per second. Still modest.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#32

Post by rlau26 »

lumpypua wrote: post #28
falsedan wrote: post #27

Earlier quoted context omitted.

> 100 million reviews/year is only 3 reviews per second on average That's writes, I imagine the reads would be a bit higher.

And ultimately, writes resulting from those reads. I imagine yelp has pretty stout analytics/metrics pipeline fed off views and conversion tracking.

This. A lot of people look at apps like Instagram and Facebook, and think that they're simple to build and manage, when they don't realize that the part of the app the consumer directly interacts with on their screen is the tip of the iceberg when it comes to the entire business.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#33

Post by NhanH »

eellpp wrote: post #31
z3t4 wrote: post #8

100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance. Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up in…

Applying the 80/20 principle: 80% of traffic occurs in the 20% of years time. Which is around 12 reviews per second. Still modest.

Not entirely on topic, but is the 80/20 principle recursive?

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#35

Post by nsp »

NhanH wrote: post #33
eellpp wrote: post #31

Earlier quoted context omitted.

Applying the 80/20 principle: 80% of traffic occurs in the 20% of years time. Which is around 12 reviews per second. Still modest.

Not entirely on topic, but is the 80/20 principle recursive?

Yes.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#36

Post by je42 »

vkou wrote: post #21
dexterdog wrote: post #7

Earlier quoted context omitted.

I think that is why the weekly number is only 50. It's a net.

Not to mention that some number of those LOC are unit tests / integration tests (Which can take more code than a feature.)

mmh. usually the LoC of unittests/integration/end2end are much more than the code under test. In my experience: between 3-6 times as much.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#37

Post by lifeisstillgood »

one of my big questions is how did you arrange security? Kerberos? What about maintaining service-users and permissions to send / rec messages, or some form of web of trust? With message based comms encryption for the intended users becomes possible?

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#38

Post by poooogles »

ricardobeat wrote: post #16
justinc-yelp wrote: post #9

Earlier quoted context omitted.

Most of the stream processing in the Data Pipeline happens inside of an internal project called PaaStorm, which is storm-like. It was built to take advantage of our platform as a service ( http://engineeringblog.yelp.com/2015/11/introducing-paasta-a... ), which handles process scheduling really well. Architecturally, it's pretty similar to Samza, with distributed processes communicating using Kafka. We do use Spark s…

Hi Justin! Thanks for sharing, very interesting stuff. How do you scale Kafka to handle the massive amount of traffic (and storage) that you seem to generate daily? With services talking among themselves via HTTP there is a lot of resilience built-in. Do you have anything in place to avoid this becoming a single point of failure? It must have become the most critical piece of your infra.

Scaling Kafka is pretty simple, the operations document contains most things you'll need to get started [1].

We push 500k documents a second through over 10 6 core/24gb ram hosts pretty uneventfully. Only real pointer is to size ZK appropriately and make sure you leave lots of memory for the file system cache.

1 - https://kafka.apache.org/documentation.html#operations

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#39

Post by TeMPOraL »

z3t4 wrote: post #8

100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance. Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up in…

Whenever I see headlines like these, "X billion messages per day/hour/second in some web service Y", I wonder what the hell is that service doing that it generates so much messages? I.e. I could understand Facebook, with its billion+ of users and built-in messaging platform, could generate billion of "messages" a day. But a mostly read-based service like Yelp?

But I finally realized - those messages are probably mostly tracking, ads, more tracking, some infrastructure work and even more ads & tracking. The sausage machine that turns people into money.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#40

Post by ju-st »

eellpp wrote: post #31
z3t4 wrote: post #8

100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance. Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up in…

Applying the 80/20 principle: 80% of traffic occurs in the 20% of years time. Which is around 12 reviews per second. Still modest.

Even a Sqlite DB could handle this
Post reply on HN