Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

311–320 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#311

Post by jameshart »

Getting everything onto one machine works great until... it no longer fits on one machine.

You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit.

Now you have to go distributed.

And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization and all of a sudden none of that matters any more.

Or you accept that there's a hard ceiling on what your system will ever be able to do.

This is like building a video game that pushes a specific generation of console hardware to its limit - fantastic! You got it to do realtime shadows and 100 simultaneous NPCs on screen! But when the level designer asks if they can have water in one level you have to say 'no', there's no room to add screenspace reflections, the console can't handle that as well. And that's just a compromise you have to make, and ship the game with the best set of features you can cram into that specific hardware.

You certainly could build server applications that way. But it feels like there's something fundamental to how service businesses operate that pushes away from that kind of hyperoptimized model.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#312

Post by icedchai »

wtallis wrote: post #255
vidarh wrote: post #237

Earlier quoted context omitted.

Biggest problem with this is the lack of considering analytics.twitter.com and ads.twitter.com. Twitter stores event data about everything that happens to a tweet, and lets you target ads with a lot of precision. While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet view…

A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.

True. You'd also save a ton of less operations and engineering staff.

Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#314

Post by icedchai »

thorncorona wrote: post #298
quickthrower2 wrote: post #196

Earlier quoted context omitted.

… HN thread that reinvents CDN …

Twitter is big on real time though. Each user gets served their own feed in real time.

This doesn't seem like an issue. You can store images / other assets on the CDN, and serve the real time stuff from the origin without caching.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#315

Post by ummonk »

britneybitch wrote: post #212
ummonk wrote: post #38

Earlier quoted context omitted.

Was it more than the salary of even a single software engineer?

We wasted 5-6 figures of salary dollars lovingly building those clusters and the automation surrounding them. We had blue-green zero downtime deploys, but no customers who would notice any downtime to begin with. I think the CTO just wanted k8s on his resume. In one afternoon (at most), I could have written a script to deploy our demo with docker compose over ssh. Sure, docker compose won't scale forever, but their r…

Fair enough. To be honest, of the abstraction tools out there, Kubernetes is the one I'd hold off on as long as possible. Load balancing proxies, docker compose, auto-scaling, cloud databases, etc. are things I'd do relatively early, but not Kubernetes.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#316

Post by eismcc »

jameshart wrote: post #311

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

You’d end up synchronizing feature releases to Moores law. Which while it sounds untenable there are large corporations that continue to use monolithic approach and vertical scaling.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#317

Post by icedchai »

pengaru wrote: post #147

This post reminds me of an experience I had in ~2005 while @ Hostway Chicago. Unsolicited story time: Prior to my joining the company Hostway had transitioned from handling all email in a dispersed fashion across shared hosting Linux boxes with sendmail et al, to a centralized "cluster" having disparate horizontally-scaled slices of edge-SMTP servers, delivery servers, POP3 servers, IMAP servers, and spam scanners. T…

Sounds reasonable. Back in the 90's, at an early ISP, we had about 6,000 POP3/SMTP accounts on a FreeBSD box with 128 megs of RAM and a decent (for 1996) SCSI disk. I recall we did some kernel tuning (max processes and file handles, maybe?)

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#318

Post by mizzao »

While you might not want to do this with actual Twitter, any sort of high-performance computing workload can run substantially faster on a single optimized machine than on a distributed computing environment.

I learned this the hard way when I was running a medium-sized MapReduce job in grad school that was over 100x faster when run as a local direct computation with some numerical optimizations.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#319

Post by robluxus »

mizzao wrote: post #318

While you might not want to do this with actual Twitter, any sort of high-performance computing workload can run substantially faster on a single optimized machine than on a distributed computing environment. I learned this the hard way when I was running a medium-sized MapReduce job in grad school that was over 100x faster when run as a local direct computation with some numerical optimizations.

Obligatory "you don't need Hadoop": https://news.ycombinator.com/item?id=14401399

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#320

Post by cellis »

untech wrote: post #286
moeny wrote: post #247

Earlier quoted context omitted.

's/(human|desktop)/smartphone/g'

Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.

Don't you need 3 numbers? Unless you believe in a flat earth ;). Also you need some slack space for metadata. Let's call it 100GB all in.
Post reply on HN