Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

101–110 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#101

Post by sroussey »

porknubbins wrote: post #69
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

I had MySQL running on some bare metal for many years without a restart.

I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully.

Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#102

Post by frmrcnck »

mongol wrote: post #76
userbinator wrote: post #19

Earlier quoted context omitted.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

If software ran without side effects, perhaps. But it doesn't. Databases grow, files are uploaded, logs pile on, messages and events propagates and filesystems fill up. This is why entropy matters.

Said differently, software with no side effects must run in an adiabatic box.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#103

Post by throwawaylinux »

emodendroket wrote: post #60
throwawaylinux wrote: post #58

Earlier quoted context omitted.

It wasn't only news outlets. A majority of my... politically-noisy tech friends on Facebook went through a recent phase of intensely posting about Twitter on fire and collapsing. Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comment…

I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office. e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.

[flagged]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#104

Post by user3939382 »

typon wrote: post #61
userbinator wrote: post #19

Earlier quoted context omitted.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

Spot on. Absolutely hate this attitude that software sitting there just gathers wear and tear as if it's a mechanical device. Software is written with a particular target platform in mind: x86, ARM, Nvidia GPUs, FPGA soft-processor etc. If the hardware you are running on doesn't change, your software should still function. If the specs of that target platform don't change, your software should still function. If the…

Unfortunately it’s not this simple because most non-trivial software is written with a dependency tree, every node of which may discover vulnerabilities (or performance problems) which, when patched, trigger update cascades in this tree.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#105

Post by hairofadog »

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is:

https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc...)

If this is the new town square, you can forward my mail to a cabin in the woods.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#106

Post by viraptor »

emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations

Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different and have different time to resolution.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#107

Post by freshpots »

throwawaylinux wrote: post #103
emodendroket wrote: post #60

Earlier quoted context omitted.

I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office. e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.

[flagged]

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#109

Post by throwawaylinux »

hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#110

Post by dcow »

porknubbins wrote: post #69
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

I have a piece of Rust software that has not gone down in its entire lifetime.
Post reply on HN