I've apologized to him before, but after having worked on Minefold for the last 2 years, I feel like I need to apologize again. The work that Jeff and the others at Twitter have done has been amazing. I was also a massive cock.
Notes on Distributed Systems for Young Bloods
11–20 of 46 posts
Re: Notes on Distributed Systems for Young Bloods
#12Re: Notes on Distributed Systems for Young Bloods
#13One of the most shameful moments of my career happened on the Saturday night before WWDC in 2010. I'd never worked on a system with more than one component or more than 20,000 lines of code. Drunk, I managed to stumble from Jeff's hot-tub to his roof and told him that Twitter's scaling problems were "stupid" and that "there were no good reasons why Twitter should go down". I've apologized to him before, but after hav…
Also, that was perhaps the biggest apologetic act I have ever witnessed...good job.
Re: Notes on Distributed Systems for Young Bloods
#141. The metrics comment could not be more true. You cannot think hard enough about the actual problems you will encounter and what metrics you will have. Have a common way to pull information about each service, and to analyze it in some common place.
2. Standardize, standardize, standardize. When someone is answering a page at 2 AM and has to deal with a component of your system that they don't really know, the more it resembles other components, the better. Think hard about how you can get every service to be written in such a way that the same critical information is available. Where is it documented? Who do I page? Where is the monitoring? Yes, random third party components won't follow your standards, and will be hard to integrate. But standardization is a good thing.
3. You need to be able to send canary requests through that trace through your whole infrastructure. You should be able to flag a front end request, and have every single request that it generates through your entire system be logged somewhere so that you can see a breakdown of what happened. Hide this so that nobody can use it to take your site down, but build the capacity for yourself. (Standardization will make such a system much easier to build!)
4. Randomly canary a small fraction of your traffic. There is tremendous value in having a random sample of traced traffic. When you're trying to understand how things work, there is nothing like taking an actual request going through a complex system, and seeing what it did. Furthermore if you've got intermittent problems for a small fraction of users, being able to look at a random slow request really, really, really helps you track down issues that otherwise would be virtually impossible to replicate.
Re: Notes on Distributed Systems for Young Bloods
#15I'm working on a mobile app that does a key exchange with a server before allowing a server-based registration or login. It's nowhere near as complex as your average distributed system.
That said, I've run into a scary amount of the things mentioned in this article in my tiny little use case. Just trying to ensure a decent user experience (timing out a comms check after two seconds rather than waiting up to 60 seconds when the phone switches from networked to disconnected) in an async message exchange needs some crazy orchestration. Keeping the code clean means refactoring stuff I thought I had nailed two months ago.
I've been programming for a long time, but this stuff humbles me. And happily, I love it.
Re: Notes on Distributed Systems for Young Bloods
#16Brilliant, made me smile :)
Re: Notes on Distributed Systems for Young Bloods
#17One of the most shameful moments of my career happened on the Saturday night before WWDC in 2010. I'd never worked on a system with more than one component or more than 20,000 lines of code. Drunk, I managed to stumble from Jeff's hot-tub to his roof and told him that Twitter's scaling problems were "stupid" and that "there were no good reasons why Twitter should go down". I've apologized to him before, but after hav…
I refer to that as the Night of a Thousand Australians. Good times.
Re: Notes on Distributed Systems for Young Bloods
#18There's a growing area where I see a lot of people starting to get involved with distributed system who didn't have to deal with it before: rich in-browser apps with their own permanent storage. Once you have a Javascript application with state speaking over one or more APIs to your backend services, you're in the domain of distributed system design. Especially if you use the application cache and support offline ope…
Re: Notes on Distributed Systems for Young Bloods
#19Intellectual honesty at its best...