The part about "Faking Data" is quite worrisome.
7 Years Of YouTube Scalability Lessons In 30 Minutes
71–77 of 77 posts
Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes
#72Uh, cheating? "Cheating - Know How to Fake Data Awesome technique. The fastest function call is the one that doesn’t happen. When you have a monotonically increasing counter, like movie view counts or profile view counts, you could do a transaction every update. Or you could do a transaction every once in awhile and update by a random amount and as long as it changes from odd to even people would probably believe it’…
No no, the correct amount of views will be recorded for a specific video, it's just that each webserver doesn't know the exact number all the time. You make each webserver fetch the correct value perhaps every hour, and fake it inbetween. You'll get an ok approximation, users can't tell the difference, and you don't have to fetch the actual number every single pageview.
Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes
#73At 11:28 he says, "at last count, there was over a million lines of python running this thing" Having never worked with code-bases larger than ~50kloc, I have a lot of trouble understanding what 1 million lines of code is needed for, especially considering that python is such a high-level language. Does anyone have any idea why there would be this much code?
that would be 1M LOC split across all the projects, libraries and modules.
Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes
#74Uh, cheating? "Cheating - Know How to Fake Data Awesome technique. The fastest function call is the one that doesn’t happen. When you have a monotonically increasing counter, like movie view counts or profile view counts, you could do a transaction every update. Or you could do a transaction every once in awhile and update by a random amount and as long as it changes from odd to even people would probably believe it’…
No no, the correct amount of views will be recorded for a specific video, it's just that each webserver doesn't know the exact number all the time. You make each webserver fetch the correct value perhaps every hour, and fake it inbetween. You'll get an ok approximation, users can't tell the difference, and you don't have to fetch the actual number every single pageview.
Sir, thank you :-) I appreciate you clarifying this!
Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes
#75"They wrote their own BSON implementation which is 10-15 times faster than the one you can download." Curious to hear more about that one. If true, I hope they open source it, because that could potentially make MongoDB a lot faster for everyone. EDIT: It's apparently in their vitess code. Relevant code: http://code.google.com/p/vitess/source/browse/#hg%2Fgo%2Fbso...
fwiw I've done some python benchmarking; it happens that I'm actually genuinely needing a faster protocol right now: http://stackoverflow.com/questions/9884080/fastest-packing-o...
Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes
#76Earlier quoted context omitted.
LOL users care and they notice a LOT..Probably something like 5% of videos have a comment about how the view count is inaccurate.
does it stop them from watching videos or in any way hinder their enjoyment of the content?
If they feel strongly enough about it to leave a comment then I think it's safe to say it does hinder their enjoyment - in the same way that obviously broken things distract and displeasure in any medium.
Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes
#77At 11:28 he says, "at last count, there was over a million lines of python running this thing" Having never worked with code-bases larger than ~50kloc, I have a lot of trouble understanding what 1 million lines of code is needed for, especially considering that python is such a high-level language. Does anyone have any idea why there would be this much code?
> Does anyone have any idea why there would be this much code? It's the world's 3rd biggest website with hundreds of billions of views and dozens of millions of users, so maybe that's why. I think it's a credit to Python that a website that does that and has grown in a fairly haphazard fashion only has about 1000k SLOCs.
Oh, it wasn't meant to be snarky or a slam -- I'm honestly just curious what kinds of things require so much code? E.g., is it one or two things that dominate usually in codebases this size, or is it just a LOT of components, each of which is tens of thousands of lines long? Do these kind of counts usually include auto-generated code?