Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

901–910 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#901

Post by Workaccount2 »

donkeyd wrote: post #846
recuter wrote: post #765

Earlier quoted context omitted.

Quoted post unavailable.

> The job cuts are clearly justified because of the extremely toxic "work culture" I'm very curious in what way the stuff Elon is doing is not creating toxic work culture? >Minimum I'm not denying there may have been toxic work culture before, but I don't see how replacing it with another form of toxic work culture is to be celebrated. Also, writing this: > There is no active independent thought around this subject t…

The 20-hour-full-time-from-home-six-figure tech job wasn't a new paradigm or a shift in c-suite work/life balance philosophy. It was 0% interest rates coupled with asset inflation leading to tech companies having more cash than they could possibly figure out how to spend. A transient financial anomaly in a whirlwind of larger economic conditions.

Now that the fed has been (rightfully)pulling the rug, people shouldn't be surprised if tech work goes back to the 60/70 hour grinds it was infamous for.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#902

Post by nobleach »

couchand wrote: post #888
nobleach wrote: post #866

Earlier quoted context omitted.

I'm still on the fence. As an engineering manager, I tend to attach faces to those "jobs". So seeing cuts, I imagine a ton of people that had to go home and tell their friends/family/etc that they no longer had a job. On the other hand. As an engineer, we tend to attach way too much self importance to our roles. Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is g…

If you think the people you work with don't have a sincere desire to accomplish things you are a terrible manager.

Nowhere do I assert that people do not have the desire to accomplish things. Perhaps you're misreading my statement about "believing the building will fall down without us" incorrectly?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#903

Post by keneda7 »

polynomial wrote: post #874
donkeyd wrote: post #830

Earlier quoted context omitted.

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.

Sensitive workplaces (which seems like most these days) have taken to calling this the Lottery Factor (as in team members quit bc they hit the lottery) to spare more delicate types the pain of imagining their peers run over in traffic accidents.

I work in the EEO/AAP space and can confirm our company calls it the Lottery Factor though it is partially because we do an office lotto pool when it gets big and jokes bled over. However it is much nicer for discussions. What will we do when employee x gets hit by a bus just does not sound good to random people walking by.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#904

Post by lupire »

throwuwu wrote: post #665
watt wrote: post #644

Earlier quoted context omitted.

Dead cat bounce. Engineers are paid $1000 a day because they create value to the tune of $10 000 a day. So maybe you saved $4 million a day in very short term, but now the momentum is gone, and soon you will not be earning $40 million a day from the value that now was not being created, just in a "few" short months. I wish Elon the best, but he could have hired his own team of "hardcore" engineers and put them to wor…

SWE are not assembly line workers, they are the people that build the automated assembly line. It’ll keep functioning well enough without them and with a much smaller team performing maintenance. I’d even bet on new feature being released faster now that they’ll be owned by highly motivated individuals instead of committees of managers and randomly assigned engs.

Yes, features are being released faster, and rolled back faster due to bugs and mismanagement, approaching infinite angular velocity of feature flips.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#905

Post by Xylakant »

polynomial wrote: post #874
donkeyd wrote: post #830

Earlier quoted context omitted.

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.

Sensitive workplaces (which seems like most these days) have taken to calling this the Lottery Factor (as in team members quit bc they hit the lottery) to spare more delicate types the pain of imagining their peers run over in traffic accidents.

Having been in the situation that a coworker did no longer arrive to work one morning because some person opened the car door when they were passing with their motorcycle, I can assure you that it's not only the more delicate type of people that all of a sudden are a bit more quiet than usual.

It's all fun and lulz until it happens to you.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#906

Post by infotropy »

skeeter2020 wrote: post #898
throw0101c wrote: post #744

Earlier quoted context omitted.

> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk. [1] https://en.wikipedia.org/wiki/Bus_factor

I know the bus factor is the morbid "how many people can get hit by a bus?" idea, but I actually like to present it as "how wide is the bus?" in terms of people being the conduits along which information and instructions flow; a wider bus provides redundancy. Aside from being more positive I think it's also more accurate for how we want teams to work. The "win the lottery and quit" metaphor is just stupid.

I usually rephrase it as “getting run over by the lottery” or “winning a bus”. Just to be different/humorous/less morbid.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#907

Post by nobleach »

adalacelove wrote: post #882
nobleach wrote: post #866

Earlier quoted context omitted.

I'm still on the fence. As an engineering manager, I tend to attach faces to those "jobs". So seeing cuts, I imagine a ton of people that had to go home and tell their friends/family/etc that they no longer had a job. On the other hand. As an engineer, we tend to attach way too much self importance to our roles. Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is g…

> Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is going to crumbl Never I have encountered an engineer that thought that.

One need look no further than the O'reilly book on Microservices. First chapter I believe. We call ourselves "engineers" but the truth is, real engineers have real consequences when their designs fail. (buildings fall, bridges fall, etc). It's not a unique take. Now I'm not going to argue about engineering vs developing vs programming as that opens a massive can of worms.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#908

Post by nobleach »

mobiledev2014 wrote: post #878
nobleach wrote: post #866

Earlier quoted context omitted.

I'm still on the fence. As an engineering manager, I tend to attach faces to those "jobs". So seeing cuts, I imagine a ton of people that had to go home and tell their friends/family/etc that they no longer had a job. On the other hand. As an engineer, we tend to attach way too much self importance to our roles. Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is g…

You got the numbers wrong and Elon fired the person who knows them...

typing on my phone.... perhaps if I had an Apple ][+

Re: Why Twitter didn’t go down: From a real Twitter SRE

#909

Post by ouid »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Of course its all bloat. Software runs on computers, not engineers. The default assumption for software is that it will go on working. The state might devolve, but the software is exactly as reliable right now as it was before. I'm no friend of Elon, and I think its hilarious to think that he can be king of twitter, but all these people talking about "code entropy" are certifiably insane.

Big tech maintains talent so that they won't use their knowledge of the system to produce an identical competitor without the technical debt or investor liability of the original.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#910

Post by gcr »

mlindner wrote: post #783
bombolo wrote: post #677

Earlier quoted context omitted.

USA at some point had an anti-missile system that needed periodical reboots because it was originally designed for short deployments, so the floating point variable for the clock would start to lose precision after a while.

Fixed point, not floating point. https://en.wikipedia.org/wiki/Fixed-point_arithmetic

Floating point clocks do lose precision after long enough time though; see https://randomascii.wordpress.com/2012/02/13/dont-store-that...

Storing floating point coordinates for example is what causes the "farlands" world generation behavior in Minecraft, for example.

Post reply on HN