Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions of Messages a Day – Yelp's Real-Time Data Pipeline

engineeringblog.yelp.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#21

Post by vkou »

dexterdog wrote: post #7
hueving wrote: post #6

Earlier quoted context omitted.

Sweet jesus. Engineers that only produce code and never delete any are like cancer.

I think that is why the weekly number is only 50. It's a net.

Not to mention that some number of those LOC are unit tests / integration tests (Which can take more code than a feature.)

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#22

Post by jonknee »

throwawaymsft wrote: post #19
jonknee wrote: post #12

Earlier quoted context omitted.

What do Doom 3 and Yelp have to do with each other?

You'd hope a CRUD app used in "hello world" tutorials ( https://www.fullstackreact.com/articles/react-tutorial-cloni... ) would be simpler than a 3d physics engine.

Displaying a list and showing a map is one feature of Yelp, but you could also say showing "hello world" in 3D text with OpenGL is a 3D graphics engine. In the real world they are both complicated things.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#23

Post by teh »

overcast wrote: post #4

"In 2011, Yelp had more than a million lines of code in a single monolithic repo" Did I read that correctly, the Yelp product is a MILLION lines of code? For comparison, the Doom 3 source code has 601,000.

It's quite easy to get to that number of lines once you have code for integrations with other products, monitoring, devops, patched versions of broken or unmaintained libraries, a growing number of tests etc. We run a relatively small product and we already have 30k lines with just two people.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#24

Post by vhost- »

artpepper wrote: post #17
blergh999 wrote: post #11

I don't understand the following: "Yelp passed 100 million reviews in March 2016. Imagine asking two questions. First, “Can I pull the review information from your service every day?” Now rephrase it, “I want to make more than 1,000 requests per second to your service, every second, forever. Can I do that?” At scale, with more than 86 million objects, these are the same thing." Who is making 1000 requests per second…

I think it's saying that a single iteration over the entire set, would translate to 1000 requests per second for a day (if done naively as one request per object). It's really talking about the N+1 problem.

This is how I read it too. I think op was taking that too literally when they were just stating that the two are essentially equivalent at that scale.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#26

Post by joevandyk »

z3t4 wrote: post #8

100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance. Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up in…

Traffic is usually not averaged out.. it's spiky.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#27

Post by falsedan »

z3t4 wrote: post #8

100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance. Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up in…

  > 100 million reviews/year is only 3 reviews per second on average
That's writes, I imagine the reads would be a bit higher.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#28

Post by lumpypua »

falsedan wrote: post #27
z3t4 wrote: post #8

100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance. Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up in…

> 100 million reviews/year is only 3 reviews per second on average That's writes, I imagine the reads would be a bit higher.

And ultimately, writes resulting from those reads. I imagine yelp has pretty stout analytics/metrics pipeline fed off views and conversion tracking.

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#29

Post by danso »

throwawaymsft wrote: post #19
jonknee wrote: post #12

Earlier quoted context omitted.

What do Doom 3 and Yelp have to do with each other?

You'd hope a CRUD app used in "hello world" tutorials ( https://www.fullstackreact.com/articles/react-tutorial-cloni... ) would be simpler than a 3d physics engine.

You realize that just because a tutorial that purports to build a Yelp clone does not mean that it will actually teach you to build the entirety of the Yelp app, right?

Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline

#30

Post by orliesaurus »

This is a nice blog, it would be nice to also read one where you explain how you "judge" whether a review is fake or not - I have heard so many times from small business owners how legitimate reviews get hidden/deleted from their page. I wonder if it's an algorithm or a "humanized" process with lots of mechanical turks :) (Not in detail of course we don't want people to game your algos)
Post reply on HN