Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

321–330 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#321

Post by Existenceblinks »

paulryanrogers wrote: post #113
Existenceblinks wrote: post #20

Earlier quoted context omitted.

Techies in tech industry are basically eating the rich .. A lot of buzzword to suck investment money in.

Or rather cloud providers are eating the rich. Techies are carrying the plates.

It's a big meal. Those old-industry tycoons (oil/real estate/finance/alcohol/export/etc.) from gold rush era have too much money. There are a few places for money to land. Cloud seems to be the first wave, social media, messaging, big data, ai, smart everything. $200k-400k + $80k-$150k ranges are like hyenas army!

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#322

Post by Dylan16807 »

KingLancelot wrote: post #231
trishume wrote: post #16

Earlier quoted context omitted.

I specifically assumed a max tweet size based on the maximum number of UTF-8 bytes a tweet can contain (560), with a link to an analysis of that, and discussion of how you could optimize for the common case of tweets that contain way fewer UTF-8 bytes than that. Everything in my post assumes unicode.

UTF-8 is 1-4 bytes per codepoint dude, not 1-2. 280 * 4 = 1120, not 560.

Twitter doesn't allow 280 of any codepoint. The more complicated ones count double against your limit.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#323

Post by threeseed »

mschuster91 wrote: post #173
trishume wrote: post #21

Earlier quoted context omitted.

My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a…

> I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Netapp is at something > 300TB storage per node IIRC, but in any case it would make more sense to use some cloud service. AWS EFS and S3 don't have any (practically reachable) limit in size.

Have you actually used EFS/S3 before ?

Because both are ridiculously slow to the point where they would be completely unusable for a service such as Twitter whose current latency is based off everything largely being in memory.

And Twitter already evaluated using the cloud for their core services and it was cost-prohibitive compared to on-premise.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#324

Post by nyanpasu64 »

jameshart wrote: post #311

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

Video games have had 3D water for decades before screen-space reflections, and many look serviceable decades later (Super Mario Sunshine looks great at 480p though dated at higher resolutions).

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#325

Post by panarky »

BeefWellington wrote: post #279
trishume wrote: post #69

Earlier quoted context omitted.

Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashta…

I think the difficult part would be that tagging and indexing the relationship between a single tweet and all of its component hashtags (which you would then likely want metrics on to avoid needing to count indexes on, etc.) is where it would really start to inflate. Another poster dug into some implementation details that I'm not going to go into. I think you could shoehorn it into an extremely large server alongsid…

Twitter has full-text search, not just hashtags.

Also, the big data storage isn't text, it's images and videos.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#327

Post by Dylan16807 »

untech wrote: post #286
moeny wrote: post #247

Earlier quoted context omitted.

's/(human|desktop)/smartphone/g'

Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.

The planet has 2^47 square meters of surface, so more like 6 bytes.

Plus you can group together people in the same area and/or sort positions as integers and store only the deltas between them, so you can probably get down to 2-3 bytes per person.

And you can get dozens of models of smartphone with 16GB of RAM right now. So there might be a gap there but it's a very small gap. The phone of tomorrow will have the RAM.

Edit: Thinking about it more, with 2^33 people and 2^47 locations the average delta would be 2^14, and it's pretty easy to guarantee that fits into 2 bytes per person. And with a more accurate world population count you'd free up at least a gigabyte for your phone to actually operate with.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#328

Post by jck »

trishume wrote: post #90
habibur wrote: post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

I agree most HTTP server benchmarks are highly misleading in that way, and mention in my post how disappointed I am at the lack of good benchmarks. I also agree that typical HTTP servers would fall over at much lower new connection loads. I'm talking about a hypothetical HTTPS server that used optimized kernel-bypass networking. Here's a kernel-bypass HTTP server benchmarked doing 50k new connections per core second…

I once built a quick and dirty load testing tool for a public facing service we built. The tool was pretty simple - something like https://github.com/bojand/ghz but with traffic and data patterns closer to what we expected to see in the real world. We used argo-workflows to generate scale.

One thing which we noticed was that there was a considerable difference in performance characteristics based on how we parallelized the load testing tool (multiple threads, multiple processes, multiple kubernetes pods, pods forced to be distributed across nodes).

I think that when you run non-distrubuted load tests you benefit from bunch of cool things which happen with http2 and Linux (multiplexing, resource sharing etc) which might make applications seem much faster than they would be in the real world.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#329

Post by toast0 »

cellis wrote: post #320
untech wrote: post #286

Earlier quoted context omitted.

Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.

Don't you need 3 numbers? Unless you believe in a flat earth ;). Also you need some slack space for metadata. Let's call it 100GB all in.

Lat long and assuming at ground gets you most of the way there.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#330

Post by threeseed »

kissgyorgy wrote: post #242

I feel like people writing posts like this never worked in a big team at a big company on a big project. It is so obviously impossible to do this and Twitter has so many more features users will never even see, but sure, re-implement it in a couple hundred lines of Rust and Twitter will be saved...

HN unfortunately has a lot of people like George Hotz.

They are knowledgeable to a certain level but they simply aren't great engineers who almost always are humble, cautious, thoughtful and respectful of the intentions behind what other engineers build.

Anyone who thinks they can jump in and replace any tech stack without an extensive deep dive of the business requirements, design decisions, cost constraints, resource limitations etc that drove the choices deserves the pain and unemployment that inevitably follows.

Post reply on HN