Live data from Hacker News

7 Years Of YouTube Scalability Lessons In 30 Minutes

highscalability.com

31–40 of 77 posts

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#31
post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

As a datapoint, we use jitter a lot in cache infrastructure at Facebook

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#32
post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

[deleted]

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#33
post #3

What's this? Python? Apache? MySQL? But I thought you had to be running beta-release key value stores, esoteric web servers, and experimental programming languages if you wanted to scale! /s

Fast, cheap, scalable: pick two.

The fastest, easiest, arguably most reliable way to scale is throwing money at it. And apparently it's easier to hire good people who know 20-year old tech rather than 3-year old.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#34
post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

On this topic, an interesting paper from 1994 on adding randomization to network traffic sources:

http://ee.lbl.gov/papers/sync_94.pdf

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#35
post #23

"Dummer code is easier to grep for and easier to maintain. The more magical the code is the harder is to figure out how it works." A nice formulation of the kind of advice I keep reading here in HN.

Complicated code that isn't easy to maintain isn't smart, it's just complicated. One thing people screw up a lot is implementing some design pattern in a way that ends up shotgunning configuration or business logic over a wide area instead of keeping it in one location and using DRY.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#36
post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

It's kind of a weird way to describe it. The better way is that you want your caching to be probabilistic rather than deterministic. In general, you want to avoid anything that would get the nodes in a distributed system to harmonize.

The other way to solve the problem though would be to handle that circumstance cleanly. There are ways to resolve a thundering herd without creating a scalability problem.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#37
post #19
post #14

Earlier quoted context omitted.

Is RPC an actual technology? I thought it was more of a protocol design pattern.

Remote Procedure Call is a design paradigm for synchronous call-and-response network communication. The Sun RPC protocol is an actual technology defined in RFC1057: http://www.ietf.org/rfc/rfc1057.txt It's not insane, though not terribly relevant in the modern world. The only common technology still using it is NFS.

That's ONC RPC. Not the only implementation. XDR is still used in places, though you're right that it isn't widely used.

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#39
post #20

Most of this is relatively straightforward and unsurprising. But the one part that grabbed me is about "jittering". They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously). This is simple en…

A nice extension of this is to use exponentially distributed delays. Then reoccurring events form poisson processes which are really easy to reason about when composed eg if you have N servers firing off events at the same exponentially distributed rate the times are distributed the same as if you had one server firing events at N x the rate and distributing jobs uniformly at random to the other servers.

http://en.wikipedia.org/wiki/Poisson_process

Re: 7 Years Of YouTube Scalability Lessons In 30 Minutes

#40
post #23

"Dummer code is easier to grep for and easier to maintain. The more magical the code is the harder is to figure out how it works." A nice formulation of the kind of advice I keep reading here in HN.

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan
Post reply on HN