Billions of Messages a Day – Yelp's Real-Time Data Pipeline
engineeringblog.yelp.com
Billions of Messages a Day – Yelp's Real-Time Data Pipeline
1–10 of 49 posts
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#2Is Yelp still using pyleus and Apache Storm? Or have they migrated to Spark and Kafka Streams?
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#3Is Yelp still using pyleus and Apache Storm? Or have they migrated to Spark and Kafka Streams?
Well, in the article they claim they're using Kafka Streams, and there's a mention of Spark in one of the diagrams.
I don't see any mention of pyleus or Apache Storm.
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#4"In 2011, Yelp had more than a million lines of code in a single monolithic repo"
Did I read that correctly, the Yelp product is a MILLION lines of code?
For comparison, the Doom 3 source code has 601,000.
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#5"In 2011, Yelp had more than a million lines of code in a single monolithic repo" Did I read that correctly, the Yelp product is a MILLION lines of code? For comparison, the Doom 3 source code has 601,000.
It's not that surprising. Some estimates (I could be way off)
100 Engineers x 50 lines of code per engineer per week x 50 weeks per year (2 week vacation) x 4 years = ~ 1 million LOC
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#6"In 2011, Yelp had more than a million lines of code in a single monolithic repo" Did I read that correctly, the Yelp product is a MILLION lines of code? For comparison, the Doom 3 source code has 601,000.
It's not that surprising. Some estimates (I could be way off) 100 Engineers x 50 lines of code per engineer per week x 50 weeks per year (2 week vacation) x 4 years = ~ 1 million LOC
Sweet jesus. Engineers that only produce code and never delete any are like cancer.
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#7Earlier quoted context omitted.
It's not that surprising. Some estimates (I could be way off) 100 Engineers x 50 lines of code per engineer per week x 50 weeks per year (2 week vacation) x 4 years = ~ 1 million LOC
Sweet jesus. Engineers that only produce code and never delete any are like cancer.
I think that is why the weekly number is only 50. It's a net.
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#8100 million reviews/year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance.
Reading between the lines it seems the problem wasn't scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up into many small services. The blog could use a more detailed description of the problem they are actually solving.
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#9Is Yelp still using pyleus and Apache Storm? Or have they migrated to Spark and Kafka Streams?
Most of the stream processing in the Data Pipeline happens inside of an internal project called PaaStorm, which is storm-like. It was built to take advantage of our platform as a service (http://engineeringblog.yelp.com/2015/11/introducing-paasta-a...), which handles process scheduling really well. Architecturally, it's pretty similar to Samza, with distributed processes communicating using Kafka.
We do use Spark streaming, and are starting to use Kafka Streams and Data Flow, where they're a better fit. I'm personally most excited about Beam/Flink. We'll probably end up replacing the PaaStorm internals with some other tool, when one with good python support matures. Beam's event-time handling and windowing seem really promising at this point. https://www.oreilly.com/ideas/the-world-beyond-batch-streami... is a great overview of the different concerns for stream processing.
Re: Billions of Messages a Day – Yelp's Real-Time Data Pipeline
#10"In 2011, Yelp had more than a million lines of code in a single monolithic repo" Did I read that correctly, the Yelp product is a MILLION lines of code? For comparison, the Doom 3 source code has 601,000.
I work on a code base based on UE4. There's well over 3million LOC.