21–30 of 32 posts
Earlier quoted context omitted.
When unexpected load causes a production outage, it's often something you wouldn't have predicted using a load test.
Sure, but I just mean sending artificial load to see where you tip over.
Changes:
- When I joined in 2015, it was already forbidden to just log text on the server. Only a small number of SREs even had access to the text logs. Any event that you cared about needed to be logged from the system described in "App-specific metrics", which was pretty easy to use. (That said, on desktop we still used logspam because when you're debugging a desktop client doing the wrong thing, sometimes the only lead you have is tracing which lines of code are being run by looking at which logs are being printed.)
- Re: shards, by the time I joined most of the mysql was replaced by a homegrown KV/graph database on top of mysql that had a fixed number of shards (something like 255).
Things that didn't change (at least by the time I left in 2020):
- Everything was in UTC, including in the UIs for everything (e.g. metrics, exceptions, crashes, etc). I ended making a GPS clock that showed UTC to put on my desk as a fun project, and I know at least one SRE who had set basically all their clocks to UTC time (e.g. phone, laptop, calendar, etc). This is in contrast to Google, where almost every system either shows your local time or "Google Standard Time" (which is just Pacific Time).
- Python was still used for virtually everything.
Earlier quoted context omitted.
Couldn't agree more! When I am forced to use a Windows environment I find myself reaching out to these evergreen tools in many situations and is frustrated by having to find workarounds to these tools that should be ubiquitous tools in any OS. Everything I learned almost 30 years ago still holds true today when it comes to the basics. And yes I know of Cygwin, MKS toolkit etc, but trying to convince corporate IT in a…
Thankfully git is everywhere these days, and git on windows comes with most of the coreutils. I hope IT isn't telling you you're not allowed to use source control. Microsoft owns GitHub after all
I was at Dropbox from 2015-2020, and it's funny how many things have changed and how many stay the same. Changes: - When I joined in 2015, it was already forbidden to just log text on the server. Only a small number of SREs even had access to the text logs. Any event that you cared about needed to be logged from the system described in "App-specific metrics", which was pretty easy to use. (That said, on desktop we st…
Unfortunately, I don't think I'd describe dbx as a place with simplicity in infrastructure during the time I worked there, but it was good to have a standard set of rules enforced everywhere.
Early attempts at what later became Chaos Monkey / Simian Army.
"One technique we repeatedly used was creating artificial extra load in the live site. For example, we would do a lot more memcached reads than necessary. Then when memcached broke, we could quickly switch off the duplicate queries and have time to come up with a solution." As much as I want to hate this, it's actually genius and elegant in its simplicity.
https://aws.amazon.com/builders-library/reliability-and-cons...
Earlier quoted context omitted.
Couldn't agree more! When I am forced to use a Windows environment I find myself reaching out to these evergreen tools in many situations and is frustrated by having to find workarounds to these tools that should be ubiquitous tools in any OS. Everything I learned almost 30 years ago still holds true today when it comes to the basics. And yes I know of Cygwin, MKS toolkit etc, but trying to convince corporate IT in a…
Sometimes nothing beats grep, but using a Windows environment I usually just go to Python to create a quick script to handle this stuff. Lots of times it is really nice to have all your "debugging scripts" in a repo anyway. Once you have that it isn't hard to set up a basic Python environment and have your scripts there.
This may be from 2012, but it's surprising just how much of the specific advice still applies. Grep, sed, awk, xargs etc are still the canonical list of shell tools for debugging, and have been for decades. Makes you realise, learning Unix tools will likely be a better long term time investment than the latest trendy framework. I wish I could go back in time and tell that to my younger self...
Couldn't agree more! When I am forced to use a Windows environment I find myself reaching out to these evergreen tools in many situations and is frustrated by having to find workarounds to these tools that should be ubiquitous tools in any OS. Everything I learned almost 30 years ago still holds true today when it comes to the basics. And yes I know of Cygwin, MKS toolkit etc, but trying to convince corporate IT in a…
Not every OS needs to be an UNIX clone.