431–440 of 544 posts
Earlier quoted context omitted.
One possibility on 5) Too many KPIs and quarter goals to be reached, too many corners cut.
Obligatory to watch with this comment: "Let's deploy to production" https://youtu.be/5p8wTOr8AbU
Earlier quoted context omitted.
You know, it would be cool if you found stats on the downtime metrics of these various high-profile recent outages, and calculated the odds of having such a cluster. Statistics is hard, though, and avoiding a "Texas Bulls-eye" would be hard.
"Celebrities die 2.7183 at a time": http://ssp.impulsetrain.com/celebrities.html
Earlier quoted context omitted.
You know, it would be cool if you found stats on the downtime metrics of these various high-profile recent outages, and calculated the odds of having such a cluster. Statistics is hard, though, and avoiding a "Texas Bulls-eye" would be hard.
Already did that here: https://news.ycombinator.com/item?id=20356610
Earlier quoted context omitted.
If the ops team got beer every time the servers went down (as a reward for fixing them) then they'd have an incentive for the servers to go down.
What the hell are all of you smoking, some moderately expensive alcohol is nowhere near enough reward to take down a service.
Earlier quoted context omitted.
To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy. That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING H…
Just a quick note not directed at OP but for any other engineers that may be unaware, these days AWS makes provisioning a read replica painless, and you can point the CEO to up-to-the-minute data while essentially firewalling the queries from customer operations.
Earlier quoted context omitted.
You can't delete that much data in a matter of seconds digitally. You need to physically harm the hardware.
Wipe the encryption key of your SSD, small amount of data to wipe, and the whole SSD is unrecoverable. FWIW at least OCZ SSDs have an encryption key flashed into it even if you didn't turn encryption on. Putting a new firmware on it can wipe this key and make your old data inaccessible. Source: got a buggy OCZ firmware that failed to make the device appear on the SATA bus, only way OCZ could fix it was to install new…
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
The fancy new feature, increasing traffic, or adding AI to something will generate headlines, accolades, and positive attention. Not having outages is something everyone expects by default. This goes double for work that prevents outages. No one wins awards for what doesn't happen.
How many medals are pinned on the guys installing fire sprinklers?
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
6) We used to have many small outages at different websites. Now, with so many things that once were separate small sites aggregated on sites like FB, Twitter, Reddit, etc we have a few large sites, so we have aggregated the failures along with that. The failure rate, by this theory, is the same, but we have replaced "many small failures" with "periodic wide-spread failures, big enough to make headlines". Turning many small problems into a few bigger ones. Just another hypothesis.
Earlier quoted context omitted.
"I would intercept these rewards and put them in my backpack for the bus ride home, in order to avoid creating perverse incentives for the operations team. But did anyone call me 'hero'?" Wait so you stole rewards for a team that was spending time (I assume extra or stressful) on something you didn't do or have any part in. And you want a cookie? I mean I get it, the company was probably not great in it's infancy. Bu…
It's a joke. Laugh, it's funny.
Earlier quoted context omitted.
>> he hit Control-C, and Twitter came back up. Monolithic architecture. When I did security work I fought this every day. Moving away from it is a nightmare of technical debt and heated debate about who should control what. I'm reminded of a story from the early days of MSN. The legend goes that in the late 90s MSN ran out of one cabinet, a single server. The server had redundant power supplies, but only one physical…
> Monolithic architecture. This particular problem had nothing to do with a monolithic architecture. Your app can be a monolith, but that still doesn't mean your BI team can't have a separate data warehouse or at least separate read replicas to run queries against.