Live data from Hacker News

Notes on Distributed Systems for Young Bloods

somethingsimilar.com

11–20 of 46 posts

Re: Notes on Distributed Systems for Young Bloods

#11
One of the most shameful moments of my career happened on the Saturday night before WWDC in 2010. I'd never worked on a system with more than one component or more than 20,000 lines of code. Drunk, I managed to stumble from Jeff's hot-tub to his roof and told him that Twitter's scaling problems were "stupid" and that "there were no good reasons why Twitter should go down".

I've apologized to him before, but after having worked on Minefold for the last 2 years, I feel like I need to apologize again. The work that Jeff and the others at Twitter have done has been amazing. I was also a massive cock.

Re: Notes on Distributed Systems for Young Bloods

#12
Having done distributed systems for many years the only thing I would like to add is that all systems are not equal so some of the things mentioned in this article might not work. It was my experience that the real key to making a distributed system work was to identify the critical paths and segment and encapsulate the functionality of different components into the smallest possible implementations. Then define robust messaging between the components to allow for coordination. In my case we were making control systems for Naval systems so we had a much different set of requirements then websites, but it was a massively distributed system ( over 10K nodes ) with a strong requirement for redundancy and latency ( no ethernet here just field bus networks ). I even wrote and published a paper on it some time back. Point is this is not the only path there are other ways to implement distributed systems and there is still much to be discovered in the field.

Re: Notes on Distributed Systems for Young Bloods

#13

One of the most shameful moments of my career happened on the Saturday night before WWDC in 2010. I'd never worked on a system with more than one component or more than 20,000 lines of code. Drunk, I managed to stumble from Jeff's hot-tub to his roof and told him that Twitter's scaling problems were "stupid" and that "there were no good reasons why Twitter should go down". I've apologized to him before, but after hav…

Until you experience the unrelenting hunger for your data by the gnawing mouths of the multitudes, it is hard to understand what a big job it is to hold them back. I have almost had panic attacks from (comparatively) much smaller user-bases than that.

Also, that was perhaps the biggest apologetic act I have ever witnessed...good job.

Re: Notes on Distributed Systems for Young Bloods

#14
Great article. There are just a few things that I would add to it.

1. The metrics comment could not be more true. You cannot think hard enough about the actual problems you will encounter and what metrics you will have. Have a common way to pull information about each service, and to analyze it in some common place.

2. Standardize, standardize, standardize. When someone is answering a page at 2 AM and has to deal with a component of your system that they don't really know, the more it resembles other components, the better. Think hard about how you can get every service to be written in such a way that the same critical information is available. Where is it documented? Who do I page? Where is the monitoring? Yes, random third party components won't follow your standards, and will be hard to integrate. But standardization is a good thing.

3. You need to be able to send canary requests through that trace through your whole infrastructure. You should be able to flag a front end request, and have every single request that it generates through your entire system be logged somewhere so that you can see a breakdown of what happened. Hide this so that nobody can use it to take your site down, but build the capacity for yourself. (Standardization will make such a system much easier to build!)

4. Randomly canary a small fraction of your traffic. There is tremendous value in having a random sample of traced traffic. When you're trying to understand how things work, there is nothing like taking an actual request going through a complex system, and seeing what it did. Furthermore if you've got intermittent problems for a small fraction of users, being able to look at a random slow request really, really, really helps you track down issues that otherwise would be virtually impossible to replicate.

Re: Notes on Distributed Systems for Young Bloods

#15
The timing of this one is scary.

I'm working on a mobile app that does a key exchange with a server before allowing a server-based registration or login. It's nowhere near as complex as your average distributed system.

That said, I've run into a scary amount of the things mentioned in this article in my tiny little use case. Just trying to ensure a decent user experience (timing out a comms check after two seconds rather than waiting up to 60 seconds when the phone switches from networked to disconnected) in an async message exchange needs some crazy orchestration. Keeping the code clean means refactoring stuff I thought I had nailed two months ago.

I've been programming for a long time, but this stuff humbles me. And happily, I love it.

Re: Notes on Distributed Systems for Young Bloods

#17

One of the most shameful moments of my career happened on the Saturday night before WWDC in 2010. I'd never worked on a system with more than one component or more than 20,000 lines of code. Drunk, I managed to stumble from Jeff's hot-tub to his roof and told him that Twitter's scaling problems were "stupid" and that "there were no good reasons why Twitter should go down". I've apologized to him before, but after hav…

This really did happen. And we're totally cool.

I refer to that as the Night of a Thousand Australians. Good times.

Re: Notes on Distributed Systems for Young Bloods

#18
post #7

There's a growing area where I see a lot of people starting to get involved with distributed system who didn't have to deal with it before: rich in-browser apps with their own permanent storage. Once you have a Javascript application with state speaking over one or more APIs to your backend services, you're in the domain of distributed system design. Especially if you use the application cache and support offline ope…

curious question: how do you get people to even click "ALLOW" when the popup says "do you want to allow this website to use localStorage"
Post reply on HN