Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

481–490 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#481

Post by MichaelZuo »

aetimmes wrote: post #477
MichaelZuo wrote: post #461

Earlier quoted context omitted.

What do you think about his interesting comment on the possibility of a mainframe? "I also didn’t try to investigate configuring an IBM mainframe, which stands a chance of being the one type of “machine” where you might be able to attach enough storage to fit historical images." It seems theoretically possible it could accomodate the entirety of Twitter in 'one machine'.

It depends on what you (or OP) mean by "one machine". There was a HPC cluster at Princeton when I worked there (which, looking at their website, has since been retired) that was assembled by SGI and outfitted with a customized Linux unikernel that presented itself as a single OS image, despite being comprised several disparate racks of individual 2-4u servers. You might be able to metaphorically duct-tape enough mach…

Perhaps the hardware cost would be higher with one big mainframe, but there could be many subtle advantages that would combine to reduce the overall cost, since personnel costs make up a huge chunk of total opex.

I don't have enough experience to say whether having the entirety of Twitter sit in one really big metal box would be perceived to be sufficiently advantageous or not.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#482

Post by jrk »

nkellenicki wrote: post #460
dmitriid wrote: post #381

Earlier quoted context omitted.

> Tweets alone generate petabytes of data a year Nope. It's not Tweets that generate that data. It's the insane amount of (mostly unnecessary) noise that gets thrown into the mix: analytics, logs, metrics, you name it. Every time you scroll Twitter sends multiple events to the server. That alone will generate a large chunk of those petabytes.

No, that's the second link - generated data, separate from tweets. Tweets alone generate petabytes of data a year. https://ankush-chavan.medium.com/twitter-data-storage-and-pr... Also, many people would disagree that stuff required to run a business is "mostly unnecessary".

No, they don't. In spite of the confusing wording in the post you cite, its petabytes/year claim is not derived from the 500m tweets/day claim – it must include metadata and/or multimedia.

This was all already derived (correctly) in the original post. Recapitulating:

500m tweets/day * (conservatively) 512B/tweet * 365 days/yr ~= 90 TiB/yr

Assuming compression and variable-length encoding of this long tail in colder storage, it's more likely Yes, this excludes analytics metadata, which as you suggest would not support Twitter's current ad products. But your core repeated claim about tweets alone is two orders of magnitude off.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#483

Post by saagarjha »

mgaunard wrote: post #407
saagarjha wrote: post #374

Earlier quoted context omitted.

> It wouldn't be surprising at all that you could build something equivalent to Twitter on just one beefy machine, maybe two for redundancy. The blog post kind of gets a very cut-down version of Twitter running on a single machine. Actual Twitter absolutely would not work.

Whatever you need to tell yourself to justify the bloat. Of course it's a proof-of-concept, it's not a drop-in replacement for Twitter.

What path do you see to scale this proof of concept to something that’s actually comparable to Twitter?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#484

Post by NavinF »

ilyt wrote: post #446
NavinF wrote: post #355

Earlier quoted context omitted.

I'll go ahead and quote that blog post because they block HN users using the referer header. --- "Federation" now apparently means "DDoS yourself." Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops responding. The server is basically unusable for 30 to 60 seconds until the stampede of Mastodons…

> I personally find this absolutely hilarious. Is that blog hosted on a Raspberry Pi or something? "Over a thousand" requests per second shouldn't even show up on the utilization graphs on a modern server. Mastodon is written on Ruby on Rails. That should really answer all your questions about the problem but if you're unfamiliar Ruby is slow compared to any compiled language, Rails is slow compared to near-every fra…

That makes sense, but I'm pretty sure jwz was whining about his blog getting DDoSed not a mastodon server.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#485

Post by deterministic »

aetimmes wrote: post #430

(Disclaimer: ex-Twitter SRE) > There’s a bunch of other basic features of Twitter like user timelines, DMs, likes and replies to a tweet, which I’m not investigating because I’m guessing they won’t be the bottlenecks. Each of these can, in fact, become their own bottlenecks. Likes in particular are tricky because they change the nature of the tweet struct (at least in the manner OP has implemented it) from WORM to wr…

What are your thoughts on the fact that Twitter is still functioning fine after thousands of engineers have left?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#486

Post by aetimmes »

deterministic wrote: post #485
aetimmes wrote: post #430

(Disclaimer: ex-Twitter SRE) > There’s a bunch of other basic features of Twitter like user timelines, DMs, likes and replies to a tweet, which I’m not investigating because I’m guessing they won’t be the bottlenecks. Each of these can, in fact, become their own bottlenecks. Likes in particular are tricky because they change the nature of the tweet struct (at least in the manner OP has implemented it) from WORM to wr…

What are your thoughts on the fact that Twitter is still functioning fine after thousands of engineers have left?

It sounds like those engineers did good work.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#487

Post by aetimmes »

MichaelZuo wrote: post #481
aetimmes wrote: post #477

Earlier quoted context omitted.

It depends on what you (or OP) mean by "one machine". There was a HPC cluster at Princeton when I worked there (which, looking at their website, has since been retired) that was assembled by SGI and outfitted with a customized Linux unikernel that presented itself as a single OS image, despite being comprised several disparate racks of individual 2-4u servers. You might be able to metaphorically duct-tape enough mach…

Perhaps the hardware cost would be higher with one big mainframe, but there could be many subtle advantages that would combine to reduce the overall cost, since personnel costs make up a huge chunk of total opex. I don't have enough experience to say whether having the entirety of Twitter sit in one really big metal box would be perceived to be sufficiently advantageous or not.

I think there are several TCO issues you'd run into here:

- vendor lock-in: anyone who has worked at a shop running Sun SPARC machines when they got purchased by Oracle can speak to the pain involved with negotiating software licenses or hardware support contracts with the Only Game In Town.

- the price/scarcity of mainframe talent: you're going to have to pry IBM z-series experts away from banks who are paying 50-100% over market rate, oftentimes in straight cash, to maintain systems that are propping up the United States economy in its entirety. Not to mention - my first job out of college >10 years ago had a mainframe, and I was incredulous that _anyone_ still had or needed one in the 21st century. Now I can appreciate the specific tradeoffs being made that caused the business to choose a mainframe, but attracting top-tier cost-effective junior dev talent out of college becomes several orders of magnitude more difficult once the word "mainframe" leaves your recruiters' lips.

- scalability: in the event that you ever decide to add features or functionality (or, say, increase your tweet character limit by an order of magnitude), you have now committed yourself to scaling your systems in units of mainframes costing millions per unit, as opposed to servers costing five figures per unit (not to mention, you probably need a dev environment that's airgapped from your prod environment, which means yet _another_ mainframe...)

- build vs. buy: using the same commodity x86_64/ARM hardware and Linux kernel that everyone else is using allows you to take advantage of all of the open-source datacenter software being built for that happy-path profile. The minute you stray from that path, the engineering-hour cost of everything you do has the potential to skyrocket, because you can't use anything off-the-shelf and need to recompile everything for z/Architecture. In fact, based on some cursory web searches, it doesn't appear that you can compile the Rust toolchain to even _run_ on z/OS as of today, so at minimum, OP would be committing to implementing that.

But at the end of the day, the constraining resource in every software organization I've encountered has been engineering hours, and by choosing a mainframe you're drastically limiting the potential number of engineering hours available to you in the employee market.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#489

Post by spullara »

steviesands wrote: post #450
spullara wrote: post #377

Earlier quoted context omitted.

The basic idea of the system was to scan a reverse chronologically ordered list of "user id, tweet id", filtering out any tweet whose user wasn't in the follow set (or sets in the case of scan sharing) until you retrieved enough tweets for the timeline request. There are a bunch of variants in the patent, but that is the basic idea. At the time, I estimated that Twitter was spending 80% of its CPU time in the DC doin…

Interesting, 80% seems a bit on the higher end nowadays though? For example, Google quantified this as the "datacenter tax" and through their cluster wide profiling tooling saw that it was 22-27% of all CPU cycles (still a huge amount). They go a different route and suggest hardware accelerators for common operations. Datacenter tax was defined as: "The components that we included in the tax classification are: proto…

This was back when there was 0 encryption, 0 compression, and using thrift and there is little actual business logic.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#490

Post by spullara »

pg314 wrote: post #403
spullara wrote: post #377

Earlier quoted context omitted.

The basic idea of the system was to scan a reverse chronologically ordered list of "user id, tweet id", filtering out any tweet whose user wasn't in the follow set (or sets in the case of scan sharing) until you retrieved enough tweets for the timeline request. There are a bunch of variants in the patent, but that is the basic idea. At the time, I estimated that Twitter was spending 80% of its CPU time in the DC doin…

Could you give an insight into the reasons that such a system never replaced the existing implementation?

It is extremely difficult to change out data formats.
Post reply on HN