Earlier quoted context omitted.
eh, it's a boring story source: work there
someone tripped over a cable?
that would be funny at least.
no, just a bad config deploy.
Twitter was down
421–430 of 544 posts
Re: Twitter was down
#422Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
Software complexity escalating over time? Please! The new microservices architecture we have been migrating to over the last year or so is so stable and makes tracking down problems a walk in the park. Not to mention the NOSQL database is a dream come true, as long as you don't need to query anything other than the partition key.
Re: Twitter was down
#423Earlier quoted context omitted.
Pretending that junior engineers is the problem, is the problem.
Just checking what your objection is. Is it that you think experience is overrated, or is it just that he was speculating without any evidence?
If you don't train new and capable engineers, you'll eventually lose talent due to attrition and retirement. Talent can be grown in-house; engineering companies are much better environments than universities to learn how to build scalable platforms. The cost of acquisition is low, too, because junior engineers can still make valuable contributions while they learn to scale their impact.
Re: Twitter was down
#424Earlier quoted context omitted.
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
lol yes, whats the quote on "Don't assume bad intention when incompetence is to blame"? After seeing how people write code in the real world, I'm actually surprised there aren't more outages.
Hanlon's Razor: https://en.wikipedia.org/wiki/Hanlon%27s_razor
"Never attribute to malice that which is adequately explained by stupidity."
Re: Twitter was down
#425Earlier quoted context omitted.
Pretending that junior engineers is the problem, is the problem.
Just checking what your objection is. Is it that you think experience is overrated, or is it just that he was speculating without any evidence?
Can't speak for OP, but I can tell you what mine is.
If you have an intern or a Junior Engineer, they should have a more senior engineer to monitor and mentor them.
In the situation where a Junior Engineer gets blamed for a screw up:
1. The Senior Engineer failed in their responsibility. 2. The Senior Engineer failed in their responsibility.
A Junior Engineer should be expected to write bad code, but not put it into production, that's on the Senior. If I hit approve on a Junior Engineer's PR, it's my fault if their code brings the whole system down. If a Junior Engineer had the ability to push code without a review, it's my fault for allowing that. Either way it's my fault and it shouldn't be any other way. It's a failure to properly mentor. Not saying it doesn't happen, just that it's never the Junior Engineers fault when it does.
Re: Twitter was down
#426Earlier quoted context omitted.
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
lol yes, whats the quote on "Don't assume bad intention when incompetence is to blame"? After seeing how people write code in the real world, I'm actually surprised there aren't more outages.
Well we have an entire profession of SRE/Systems Eng roles out there that are mostly based on limiting impact for bad code. Some of the places I've worked with the worst code/stacks had the best safety nets. I spent a while shaking my head wondering how this shit ran without an outage for so long until I realized that there was a lot of code and process involved in keeping the dumpster fire in the dumpster.
Re: Twitter was down
#427Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
it's end of half. everyone is entering reviews. gotta deliver.. somerthing...
Re: Twitter was down
#428Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
since all of them happen in high profile business hours, i'd guess either #1 or #5.
For #4 to be the actual cause, outages out of business hours would be more prevalent and longer.
Re: Twitter was down
#429Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
My guess is its a slow news time of year coupled with more usage of cloud services which means these types of stories are higher profile.
Re: Twitter was down
#430Earlier quoted context omitted.
One possibility on 5) Too many KPIs and quarter goals to be reached, too many corners cut.
Obligatory to watch with this comment: "Let's deploy to production" https://youtu.be/5p8wTOr8AbU
You know, I've watched a few of those memes over the past, but this one was especially well done, and timed perfectly with his gestures even!