Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

401–410 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#401

Post by bmn__ »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

> What exactly where these thousands of employees actually doing

https://nitter.lacontrevoie.fr/libsoftiktok/status/158539526...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#402

Post by mirzap »

poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#403

Post by donkeyd »

randmeerkat wrote: post #336
ezoe wrote: post #246

Earlier quoted context omitted.

Redundancy. Now the systems are stable but human workers either be sick, leave, or die eventually. Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for…

> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…

> The only reason people work is literally for money

Theories like Hertzberg exist for a reason and show that money isn't even close to the most important thing for people to work for. If the money is good enough, there are many other factors that are way more important. Not enough money is a reason for people to leave, getting more money is hardly ever a reason to work harder. The possibility of getting more money if they work harder is a good motivator though, once they have it that stops.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#404

Post by aeyes »

TranquilMarmot wrote: post #375
drdrey wrote: post #162

Earlier quoted context omitted.

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough. The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier…

You don't even need that, the kernel OOM killer would take care of this eventually. Unless its something like Java where the garbage collector would begin to burn CPU.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#405

Post by bigiain »

poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

New owner: "Nothing ever goes wrong with the cache! It just works, look at the status logs. Why are we even paying those guys to look after it!"

Also new owner: "What even is Mesos? Why are we running something called Aurora? Obviously pure bloat. Fire the lot of them."

Re: Why Twitter didn’t go down: From a real Twitter SRE

#406

Post by henrikschroder »

mirzap wrote: post #402
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

Quoted post unavailable.

Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!"

Stop worshipping billionaires.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#407

Post by mirzap »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Yeah, and they were even profitable with ~3k employees. Then the hiring spree started and they went negative. Even if there wasn't Musk they would have to let go at least 30% of the people.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#408

Post by Karrot_Kream »

mradek wrote: post #158

Tangent but I do hope that Musks trim down results in orgs that have less “executives” and a layered cake of a org structure, and more autonomous small teams that execute on shared overarching initiatives. I really don’t understand why so many tech companies have like 8 layers of engineering levels. If the argument is that you need more money so more levels, just have a bigger band. Don’t chase titles they don’t mean…

I work at a Big Tech adjacent (or Big Tech, depends on your definition) company and was there pre-IPO and it happens for the reason that varjag described in their sibling comment. What makes it even more idiotic is, as the pre-IPO/lean culture dilutes, more people will use their level to pull rank in meetings, as the level itself becomes more of a target than the work/goals. Then the politics around levels will become ultra-competitive. Google didn't get to their tortuous promo process in a vacuum after all.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#409

Post by twblalock »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#410

Post by summerlight »

powerapple wrote: post #391
summerlight wrote: post #348

Earlier quoted context omitted.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

I think the opposite. Many softwares at its best when the team was small. Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth. It is not necessarily good for the product, actually many times, it hurts the product, but overall it is good for the company, the company will enter new areas, can explore new things. What Twitter is doing…

> Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth.

I don't think you have a good understanding on how those companies are growing and scaling out. Don't take growth for the granted. "Right product" or "Right technology" won't give you that. It only comes from solving thousands of very specific, never-ending customer problems. If you do B2B, you need to spend most of your time on very specific requests from priority customers. And they are not one, but hundreds of them if you targets $xB business. It's just physically impossible to keep up with a small team even with a very aggressive prioritization.

Still not convinced? Google has a notoriously bad reputation for their customer supports and it's primarily because of their tendency of keeping "inessential headcounts" low as possible. And think about how many cloud customers they lost to AWS and Azure. TK came to GCP and his first work was adding an army of sales and account managers. This almost immediately yielded a rapid acceleration of the platform, although it's too late to catch up.

Post reply on HN