Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

491–500 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#491

Post by MichaelZuo »

aetimmes wrote: post #487
MichaelZuo wrote: post #481

Earlier quoted context omitted.

Perhaps the hardware cost would be higher with one big mainframe, but there could be many subtle advantages that would combine to reduce the overall cost, since personnel costs make up a huge chunk of total opex. I don't have enough experience to say whether having the entirety of Twitter sit in one really big metal box would be perceived to be sufficiently advantageous or not.

I think there are several TCO issues you'd run into here: - vendor lock-in: anyone who has worked at a shop running Sun SPARC machines when they got purchased by Oracle can speak to the pain involved with negotiating software licenses or hardware support contracts with the Only Game In Town. - the price/scarcity of mainframe talent: you're going to have to pry IBM z-series experts away from banks who are paying 50-10…

That's the tradeoff I was referring to.

Whether or not removing several intermediary layers of abstraction, and the commensurate 100x (?) boost in efficiency, between users tweeting a hashtag and the actual electrons vibrating, is worth taking on the significant constraints you've enumerated.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#492

Post by adam_arthur »

TacticalCoder wrote: post #244

TFA, to me, touches about something I've wondered about a very long time ago: what are the implications of CPU and storage growing at much faster rates than human population? Back in the 486 days you wouldn't be keeping, in RAM, data about every single human on earth (let's take "every single human on earth" as the maximum number of humans we'll offer our services to with on our hypothetical server). Nowadays keeping…

The implication is that scalability problems in software will get easier and easier over time, and far fewer developers will be needed to maintain these systems.

Which is already largely true today with the advent of serverless. Most maintenance work can center around application logic rather than scaling physical machines/maintaining versioning.

It's clear that many modern applications would take an order of magnitude more people to run even just 20 years ago. That trend will only continue

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#493

Post by hinkley »

Hasu wrote: post #277
hinkley wrote: post #232

Earlier quoted context omitted.

Yes, but it won't be because they're better, it'll be because they're different. Like last time, and the time before that, and the time before that, and the time before that.

You're right, no progress has ever been made in software, no new ideas are better than any old ideas, and it's fads all the way down. The only difference between software today and software in 1980 is that today's software is hip and software from 1980 is square. I understand the frustration with flavor of the week "best practices" and the constant churn of frameworks and ideas, but software engineering as a practice…

Motion is not progress.

In fact it’s often meant to distract from the lack of progress.

It’s not the lack of progress that is the concern, it’s the subterfuge.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#494

Post by _zachs »

aetimmes wrote: post #430

(Disclaimer: ex-Twitter SRE) > There’s a bunch of other basic features of Twitter like user timelines, DMs, likes and replies to a tweet, which I’m not investigating because I’m guessing they won’t be the bottlenecks. Each of these can, in fact, become their own bottlenecks. Likes in particular are tricky because they change the nature of the tweet struct (at least in the manner OP has implemented it) from WORM to wr…

Thanks for the insight! At a high-level, how did Likes work when you were at Twitter? Were a certain amount of Like requests batched then applied at the DB level at the same time to ease writes?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#495

Post by mschuster91 »

threeseed wrote: post #323
mschuster91 wrote: post #173

Earlier quoted context omitted.

> I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Netapp is at something > 300TB storage per node IIRC, but in any case it would make more sense to use some cloud service. AWS EFS and S3 don't have any (practically reachable) limit in size.

Have you actually used EFS/S3 before ? Because both are ridiculously slow to the point where they would be completely unusable for a service such as Twitter whose current latency is based off everything largely being in memory. And Twitter already evaluated using the cloud for their core services and it was cost-prohibitive compared to on-premise.

EFS indeed sometimes has latency issues I've never been able to track down, but S3 with Cloudfront? That is more than enough as a CDN.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#496

Post by jwmoz »

saagarjha wrote: post #366
BeefWellington wrote: post #271

Earlier quoted context omitted.

I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. I agree mostly. Where I differ in that I would argue that hashtags were THE thing that Twitter is most known for but that could be a perspec…

FWIW my opinion of hashtags (relatively new to Twitter) is that they’re only used by brands and mostly cringe people

Originally they were the main feature I'd say. Now they've been deprecated by the algorithm that surfaces and recommends stuff.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#497

Post by Nextgrid »

drowsspa wrote: post #220
mgaunard wrote: post #203

All web and cloud technologies are inherently inefficient, and most programmers don't know networking or even how hardware works sufficiently well to optimize for high througput and low-latency. There was an article just yesterday about how Jane Street had developed an internal exchange way faster than any actual exchange by building it from the ground up, thinking about how the hardware works and how agents can inte…

I'm always reminded of that tweet from @SwiftOnSecurity https://mobile.twitter.com/SwiftOnSecurity/status/1485822027... > Once you understand your computer has 16 cores running at 3GHz and yet doesn't boot up in .2 nanoseconds you understand everything they have taken from you. With their infinite VC money at their disposal, and with their programmers having 100 GHz machines with thousands of cores, 128 TB of RAM and…

It's less about the lack of incentive to reduce bloat and more about the incentive to create bloat in order to justify one's position and pad the resume for the next positions.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#498

Post by Sohcahtoa82 »

jrk wrote: post #482
nkellenicki wrote: post #460

Earlier quoted context omitted.

No, that's the second link - generated data, separate from tweets. Tweets alone generate petabytes of data a year. https://ankush-chavan.medium.com/twitter-data-storage-and-pr... Also, many people would disagree that stuff required to run a business is "mostly unnecessary".

No, they don't. In spite of the confusing wording in the post you cite, its petabytes/year claim is not derived from the 500m tweets/day claim – it must include metadata and/or multimedia. This was all already derived (correctly) in the original post. Recapitulating: 500m tweets/day * (conservatively) 512B/tweet * 365 days/yr ~= 90 TiB/yr Assuming compression and variable-length encoding of this long tail in colder s…

> 500m tweets/day * (conservatively) 512B/tweet * 365 days/yr ~= 90 TiB/yr

I wonder if the "Petabytes" figure being claimed includes pictures/videos that can be attached to a Tweet. In that case, I could easily see "Petabytes/year" be accurate.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#499

Post by nimish »

threeseed wrote: post #476
nimish wrote: post #353

Earlier quoted context omitted.

Great, staple a few ML accelerators to your NIC. Nvidia sells them! You could build an entire supercomputer style setup 100% optimized for Twitter data movement and computation with COTS hardware IMO. I strongly doubt that entire datacenters need to be used if and only if Twitter obsessively optimized for hardware usage efficiency over everything else. In reality they don't and make some pretty big compromises to act…

a) No one has said that Twitter doesn't use ML accelerators. b) No one has said Twitter operates entire data centres. c) You need more than just NICs and ML accelerators to built a Twitter timeline. You need to rank the content, determine appropriate ads and combine them together. You can't do that in your network card.

Good news, Nvidia will now sell you a GPU+NIC+CPU with competitive performance: https://developer.nvidia.com/blog/accelerating-data-center-a...

Twitter operates a handful of datacenters because their scale is such that it makes sense.

Post reply on HN