Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

941–950 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#941

Post by moonchrome »

PunchTornado wrote: post #843
moonchrome wrote: post #468

Earlier quoted context omitted.

Except in these firing sprees it's not competent employees that remain - those people are usually the first to jump ship because they have options and see where it's headed. It's not like they get raises for staying to get a part of the savings from the cuts - usually they get pay freezes. Restructuring is done on whatever idea the new owner(s) have in plan - which could be equally disconnected from "real product". I…

You realise how annoying it is when you see people around you in the company making as much money as you and doing nothing useful to actually generate revenue. Data scientists that write a blog post about the most used hashtag or emoticon. Or PMs that we don't need... I'm staying and I'm glad the teams are trimmed.

>You realise how annoying it is when you see people around you in the company making as much money as you and doing nothing useful to actually generate revenue.

Absolutely. I also realize that when a company reaches the firing squad stage it's not a good idea to stick around - better polish up the CV and move to greener pastures. I have no insider information - they could be offering generous (and credible) retention bonuses for critical staff, but I've seen this happen twice before indirectly (I was gone before it started happening) and the people that stayed didn't have a good time, were lied to and left in the end anyway in a worse situation (or waited trough to bankruptcy and had to sue for outstanding wages in first place I worked in)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#942

Post by AtNightWeCode »

Anybody working in the biz knows about technical debt. At Twitter that will skyrocket now. My firm believe and experience tells me that keeping everything somewhat up to date over time always pays off.

The problem is not that things will break. Things break ALL the time. The problem is more about avoiding cascading effects and the time it takes to fix stuff.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#943

Post by hnbad »

ddalex wrote: post #645
hnbad wrote: post #604

Earlier quoted context omitted.

Leveraged buyouts are certainly a thing that sounds like it shouldn't be possible the first time you hear of it. It seems extremely odd that you can buy a company with money you don't have and then have the money take on that debt instead of having to take it on yourself. As I understand it, this works by rounding up potential loans, approaching the board of the company and getting them to sign over ownership of the…

The question is who is left holding the bag? If the stakeholders get paid, they are happy. The new owner is debt free, and have an ownership, so that's good. It seems to me that it may go two ways: either the lenders will be left holding the empty bag, if they can't make the money back - and it's their fault for doing a bad due diligence; OR the company gets sold off piece by piece to satisfy creditors (if they can't…

This view is sufficient if you view corporations entirely as entities that serve to create profit. But this view ignores externalities which are not part of this mechanical description. Specifically, what perceived value the corporation provides that allows it to generate that profit. In Twitter's case its monetization has primarily been ads, so I'm referring to what makes people invested enough in the service to make it worth for advertisers to pay to show ads to these people.

But of course your view is what's reflected in law: the corporation exists to generate profit for its shareholders, so if killing and selling it for scrap (whether directly or by proxy in a leveraged buyout ending in a firesale) provides more benefit to its shareholders than keeping it operating at a small profit, that's the logical decision. The corporation is not really "a person", it's a vehicle for its stakeholders (or shareholders). A stable service puttering along without making big profits or losses is considered the bad ending.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#945

Post by poulsbohemian »

rjzzleep wrote: post #429
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

>The reality is most huge companies are majority bloat.

Something I learned early in my career is that some companies consider this a feature not a bug. As in, they are hoarding talent so that they don't go to work for their competitor.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#946

Post by lawrenceyan »

karpathy wrote: post #539
memish wrote: post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

This is why I still regularly read HN. I appreciate your commentary.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#947

Post by fragmede »

john_the_writer wrote: post #712
b3lvedere wrote: post #559

Earlier quoted context omitted.

Whenever clients complain about those costs and efforts, i tell them it's just like their car. Your car is working perfectly fine so why should you pay for maintenance?

software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly.

Software bit-rots. That app from 1998 doesn't interact with today's world. As the world evolves around us, needs change and software has to change to keep up. That's not to say there aren't companies out there that rely on some ancient Windows 98 software program running on similarly ancient hardware, because there are. But Twitter as a piece of software isn't some static thing. It's needs are constantly changing and the software has got to keep up.

Your PalmOS app doesn't run on any modern hardware except under emulation. (Which is sad, I loved my Centro and held onto it for as long as I could.) The last release of PalmOS was in 2007, 15 years ago. Most hardware from that long ago is dead, and thus your software is dead. broken down by entropy to the hardware.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#948

Post by mtejo »

leftcenterright wrote: post #510
mtejo wrote: post #2

Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!

Kudos for nice work! What did you make of Mudge's report regarding resiliency of data-centers? > Insufficient data center redundancy, 59 without a plan to cold-boot or recover from even minor overlapping data center failure, raising the risk of a brief outage to that of a catastrophic and existential risk for Twitter's survival. - https://techpolicy.press/wp-content/uploads/2022/08/whistleb...

Thanks! There was a lot in that and I didn't follow it closely. Honestly that always confused me about the dc failing prep. Like I get that it would be a good idea to have a documented plan but at the same time, we did lose a whole DC and figured out how to recover(not that it was that easy) Im not sure having a cold boot prep plan would have helped that much. The things that failed that I dealt with personally during that event, I don't know if I would have foreseen. The site went on afterwards so it didn't seem to be existential.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#949

Post by chasd00 »

pclmulqdq wrote: post #788
birdyrooster wrote: post #567

Earlier quoted context omitted.

They will stagnate and have more failed launches than Space X.

There's something poetic about NASA sending a rocket to the moon at the same time as Elon Musk is fucking around with Twitter instead of sending a SpaceX rocket there too.

Even in all the complexity of spaceflight, Twitter is the single greatest risk to SpaceX's mission. I would laugh but i'm crying.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#950

Post by MockObject »

spbaar wrote: post #834
chrsig wrote: post #634

Earlier quoted context omitted.

advertisers wont care in a couple months. the situation has lots of public attention now, which is what the advertisers are actually afraid of. They don't want their brand associated with the craziness. They don't actually care one way or the other about any choice musk makes. They're not going to just walk away from a 300+ million person audience permanently just due to principles

They might if other channels aren't saturated and things are still in flux. Kanye was unbanned yesterday. Do you think he's going to behave better or worse than before? If he behaves worse then that's another headline, another news cycle about whether he should be banned or not. Right now the trend is advertisers getting more spooked because of unpredictability and Musk being unable to reassure them in calls he has b…

Musk says he had no role in bringing Kanye West back on Twitter

https://www.reuters.com/world/us/kanye-west-back-twitter-aft...

Post reply on HN