Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

7 Years Of YouTube Scalability Lessons In 30 Minutes

highscalability.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#51

Post by cpeterso »

ajross wrote: post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

A counterexample is that the Linux kernel tries to schedule timer events for the same deadline time. That allows the processor to sleep longer because the kernel doesn't need to wake up as often just to handle 1 or 2 timer events.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#52

Post by supersillyus »

hendzen wrote: post #26

"They wrote their own BSON implementation which is 10-15 times faster than the one you can download." Curious to hear more about that one. If true, I hope they open source it, because that could potentially make MongoDB a lot faster for everyone. EDIT: It's apparently in their vitess code. Relevant code: http://code.google.com/p/vitess/source/browse/#hg%2Fgo%2Fbso...

I believe it could be faster still. Their Itoa cache is a map[int]string, where it could be a []string. Also, I suspect that a few more primitive type special cases in the first type switch in EncodeField could go a long way.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#53

Post by ilaksh »

cwbrandsma wrote: post #2

I love the part on faking data. I take the viewpoint that only software testers care that the comment count is exactly correct in the majority of system. Users don't care.

LOL users care and they notice a LOT..Probably something like 5% of videos have a comment about how the view count is inaccurate.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#54

Post by moe »

cbsmith wrote: post #12
pothibo wrote: post #5

I fully agree with Youtube faking data. However, I reckon they are faking a bit too much. Many times I would see 2000 likes and the video having 1700 views (Viral videos that is). I knew the view counter wasn't propagated but the likes were and I was like: "Damn this is Youtube, kinda disappointing..." I guess if both were propagated at the same time I wouldn't mind.

I honestly don't understand why they simply don't use out of sync data. You could have nodes periodically send aggregates of likes & views, and then add those in to the total ever N heartbeats. Why bother fudging the in-between.

They're probably propagating the likes and views independently. Which still doesn't explain why they allow counter-intuitive gaps like that instead of fixing them up on the client-side in javascript.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#55

Post by sk5t »

cpeterso wrote: post #49
ajross wrote: post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

The Adblock Plus blog details the thundering herd problems they faced. Their ad blocking lists checked for updates every 5 days. Eventually, many users' update schedules would converge on Mondays because office computers did not run over the weekend. Updates that had been scheduled for Saturday or Sunday would spill over to Monday. https://adblockplus.org/blog/downloading-a-file-regularly-ho...

Windows does much the same thing w.r.t. policy refresh (which sucks down files from domain controllers) and update of "occasionally updated" metadata like last logon timestamp.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#56

Post by kami8845 »

ilaksh wrote: post #53
cwbrandsma wrote: post #2

I love the part on faking data. I take the viewpoint that only software testers care that the comment count is exactly correct in the majority of system. Users don't care.

LOL users care and they notice a LOT..Probably something like 5% of videos have a comment about how the view count is inaccurate.

does it stop them from watching videos or in any way hinder their enjoyment of the content?

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#57

Post by mbell »

My biggest gripe with youtube: why are comment almost always repeated? Yea i realize that most you tube comments are relatively worthless but I do tend to speed through them to get a feel for what the response is to a particular video. Inevitably I get through 20 comments and then the same 20 are repeated over again, often they are repeated several times. Perhaps they are trying to give the illusion of lots of comments or assuming the comments don't matter. Personally I find it extremely annoying, I'd rather them block to load more than repeat.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#58

Post by jcromartie »

What about this:

> The number of videos has gone up 9 orders of magnitude and the number of developers has only gone up two orders of magnitude.

2 orders of magnitude means at the very least, going from 9 to 100 developers, which is a huge increase, but it could mean way more. I wonder how big the team really is, and what the changing team dynamics are like on that scale at that pace.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#59

Post by modeless »

pothibo wrote: post #5

I fully agree with Youtube faking data. However, I reckon they are faking a bit too much. Many times I would see 2000 likes and the video having 1700 views (Viral videos that is). I knew the view counter wasn't propagated but the likes were and I was like: "Damn this is Youtube, kinda disappointing..." I guess if both were propagated at the same time I wouldn't mind.

Yeah, the view counter is a clear case where they've gone too far. People care about view counts and it seems like every other video I watch has people complaining in the comments about the view count being obviously wrong.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#60

Post by minimax »

breidh wrote: post #34
ajross wrote: post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

On this topic, an interesting paper from 1994 on adding randomization to network traffic sources: http://ee.lbl.gov/papers/sync_94.pdf

Fascinating paper, even if it is 18 years old now. Thanks.
Post reply on HN