Earlier quoted context omitted.
> But real world conditions can force code changes Security fixes.
Wasn't last year where a bug in a Java well-used lib (and pretty easy to use) caused mayhem in every SRE/SEC services in every banks? It was a fun two weeks. I worked like 50 hours the first three days.
Why Twitter didn’t go down: From a real Twitter SRE
781–790 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#782Earlier quoted context omitted.
That's tiny bit of an exaggeration, maaaybe. Maybe completely prophetic, though! That said, Instagram was run by just a dozen people back then, while it had hundreds of millions of users, right? So it's not a new data point. Storing, retrieving, indexing, managing 280 char blobs (with links, threads, embeds) is not exactly the most hardcore of a problem domains. Microblogs are the typical tutorial topic, and twitter'…
Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.
Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that. The followers are subscribers. They hardly even see most tweets they subscribe to as the whole thing is quite ephemeral. Same as never reading all your emails (very few people are inbox zero freaks).
The timelines are just that, an email inbox. It is very soft "real-time" at best.
Tweets older than 48 hours can just be archived to a blob store and served as a static website. Most people consume it as such, logged out and from the browser.
You think brainyquote.com is hard to scale? Think of twitter.com as unbrainyquote.com.
All the other froufrou built on top of it is more complicated but 2007 style twitter is dead simple. That is its (completely accidental) genius.
There is no shortage of systems that are x10-100 times more "web scale".
Re: Why Twitter didn’t go down: From a real Twitter SRE
#783Earlier quoted context omitted.
We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough. The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier…
USA at some point had an anti-missile system that needed periodical reboots because it was originally designed for short deployments, so the floating point variable for the clock would start to lose precision after a while.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#784It's a little thing, but it's the little things that cascade into big things that require teams of people to fix. Here, we see this person who's quite proud of his automations making the simple, but harmless error of using a possessive instead of a plural. Humans are good at reading through this sort of thing.
Computers are not. Somewhere deep in all his scripts is a misplaced ', `, or ’ waiting to break something.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#785Earlier quoted context omitted.
Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…
I think what upsets a lot of the Silicon Valley types here on HN is that people just like them are being called on their bullshit and fired en-masse for it. That has got to be uncomfortable. You know the old saying: "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It." That's what's happening here. That engineer that was humiliated publicly for defending a slow-…
If Elon had joined Twitter, and spent time understanding the business and environment and then excised the people he felt weren't contributing towards his vision, that would be one thing... but he has made arbitrary judgements based on absurd metrics like lines of code or willingness to show up at 1am to draw on a whiteboard, he has not made judgements based on the quality of the work or the value people have delivered.
Likewise, to suggest that a software engineer is bad because they were a part of a team that built a "...slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine..." is absurd: what if that person was the only reason that it wasn't 10x slower? What if, they were the lynchpin in that team ensuring that brought everyone else up to a much higher standard which ensured that what they built was usable (even if it was bad)? You cannot judge the contribution of an individual without considering the wider context.
I have no problem with a company cutting most of their software engineers (I encourage clients to minimise their exposure to software engineers, I encourage careful hiring over volume) but what Elon is doing is... not that.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#786Earlier quoted context omitted.
advertisers wont care in a couple months. the situation has lots of public attention now, which is what the advertisers are actually afraid of. They don't want their brand associated with the craziness. They don't actually care one way or the other about any choice musk makes. They're not going to just walk away from a 300+ million person audience permanently just due to principles
No but if there is no content moderation and Twitter becomes even more of an open cesspit than it already was, announcers are going to stay away because most of them do not want to be associated with toxicity. Perhaps politically active organizations will want to be there (e.g. the NRA or the ACLU), but nothing family-friendly (e.g. Disney) and most likely nothing Christmas-y during Christmas season.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#787Earlier quoted context omitted.
Software engineers at Twitter got used to working on an money losing company for 10 years and being told they’re great at their job. Then they were fired en masse because someone’s called them out on it. If your company is losing money all this time you are likely to be fired eventually in the real world. Job security in sw world had become so high that no one seemed to expect it. Everyone assumed “sure we’re losing…
This is factually untrue though. Twitter was making money in 2018 and 2019 (to the tune of ~1.2B/year in net profit out of ~3B revenue, which is a fairly high profit margin) they lost quite a bit of money in 2020 and less in the years thereafter. However, even in the years where they had negative net income EBITDA remained positive suggesting the losses were upfront investments that would be expected to be amortized…
Re: Why Twitter didn’t go down: From a real Twitter SRE
#788Earlier quoted context omitted.
> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…
They will stagnate and have more failed launches than Space X.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#789I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#790Earlier quoted context omitted.
If you consider that engineers cost about $1000 a day and he got rid of 4,000 of them… oddly it works out to about $4 million per day. I’m not an expert in math but it’s seems pretty possible. Plus they’ll have a lot more impressions to sell now that people are allowed to speak. Ad rates might drop but then someone will write an article about how they are getting great CPC on Twitter and everything will be back to no…
Dead cat bounce. Engineers are paid $1000 a day because they create value to the tune of $10 000 a day. So maybe you saved $4 million a day in very short term, but now the momentum is gone, and soon you will not be earning $40 million a day from the value that now was not being created, just in a "few" short months. I wish Elon the best, but he could have hired his own team of "hardcore" engineers and put them to wor…
If this was true then either the non-engineers in twitter have been destroying many times their salary in value or twitter has been obscenely profitable these last few years.