Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

211–220 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#211

Post by walrus01 »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

> I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform

I really wonder how much of a challenge this is and how much it occupies, not even talking about disk, but continuing the theoretical exercise in the linked URL, you can get 1U size servers with 2TB of RAM these days.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#212

Post by britneybitch »

ummonk wrote: post #38
britneybitch wrote: post #15

> colo cost + total server cost/(3 year) => $18,471/year Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.

Was it more than the salary of even a single software engineer?

We wasted 5-6 figures of salary dollars lovingly building those clusters and the automation surrounding them. We had blue-green zero downtime deploys, but no customers who would notice any downtime to begin with. I think the CTO just wanted k8s on his resume.

In one afternoon (at most), I could have written a script to deploy our demo with docker compose over ssh. Sure, docker compose won't scale forever, but their runway didn't last forever either.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#214

Post by api »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation.

Adding images, videos, other large attachments, rich search, and all the advertising and billing and analytics stuff would blow this out of the water, but... maybe not by as much as people think...? I would not be surprised if a very performance-engineered version of Twitter could run on a few dozen racks full of beefy machines like this with HPC-grade super-fast fabric interconnects.

I have a strong sense that most large scale systems are way less efficient than what's possible. They trade development ease, modularity, and velocity for performance by using a lot of microservices, flabby but easy and flexible protocols (e.g. JSON over HTTP), slow dynamic languages, and layers of abstraction that could be integrated a lot more tightly.

Of course that may be a rational trade-off if velocity, flexibility, and labor costs and logistics matter more than hardware, power, or data center floor space costs.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#215

Post by therealdrag0 »

mgaunard wrote: post #203

All web and cloud technologies are inherently inefficient, and most programmers don't know networking or even how hardware works sufficiently well to optimize for high througput and low-latency. There was an article just yesterday about how Jane Street had developed an internal exchange way faster than any actual exchange by building it from the ground up, thinking about how the hardware works and how agents can inte…

It’s “easy” to optimize for speed when you build from the ground up with no real customers or feature requirements that you can’t just conveniently ignore.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#216

Post by dpkirchner »

sayrer wrote: post #137
trishume wrote: post #121

Earlier quoted context omitted.

I think I'm pretty careful to say that this is a simplified version of Twitter. Of the features you list: - spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution…

"web previews: I'd do this by making it the client's responsibility." Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering ( https://www.jwz.org/blog/2022/11/mastodon-stampede/ ) For blocks/mutes, you have to account for retweets and quotes, it's…

FWIW, jwz uses referer checking to redirect links from HN ... for "DoS" reasons.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#217

Post by eddsh1994 »

threeseed wrote: post #118
eddsh1994 wrote: post #68

Earlier quoted context omitted.

Split the DB from the app and replicate with a load balancer?

That database is going to need to be clustered as well for resiliency. Sounds like you already have quite a number of different containers already.

Four vms isn’t too bad to manage, I was the first dev at a startup that went prelaunch to acquisition from 2 App/DB servers in EC2 for a mobile app that hit #1 on both stores with a fairly high user tx rate and some ML stuff.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#218

Post by fleddr »

yodsanklai wrote: post #168
fleddr wrote: post #158

"Through intense digging I found a researcher who left a notebook public including tweet counts from many years of Twitter’s 10% sampled “Decahose” API and discovered the surprising fact that tweet rate today is around the same as or lower than 2013! Tweet rate peaked in 2014 and then declined before reaching new peaks in the pandemic. Elon recently tweeted the same 500M/day number which matches the Decahose notebook…

> Musk bought garbage for top dollar Totally out of topic here, but could be he just wants the ability to amplify his own ideas. Also, why measure Twitter value (arbitrarily?) by number of unique tweets, rather than by read tweets?

Doing irrational things for kicks is very much on brand for Musk, but in this case he did want to back out of the deal, but couldn't.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#219

Post by tpmx »

betimsl wrote: post #213

I'm just curious as to what kind of motherboard this personal computer is going to have? I'm asking this because of the limit on PCIe bandwidth. 100gbit NIC? How?

Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf]

https://news.ycombinator.com/item?id=32519881

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#220

Post by drowsspa »

mgaunard wrote: post #203

All web and cloud technologies are inherently inefficient, and most programmers don't know networking or even how hardware works sufficiently well to optimize for high througput and low-latency. There was an article just yesterday about how Jane Street had developed an internal exchange way faster than any actual exchange by building it from the ground up, thinking about how the hardware works and how agents can inte…

I'm always reminded of that tweet from @SwiftOnSecurity https://mobile.twitter.com/SwiftOnSecurity/status/1485822027...

> Once you understand your computer has 16 cores running at 3GHz and yet doesn't boot up in .2 nanoseconds you understand everything they have taken from you.

With their infinite VC money at their disposal, and with their programmers having 100 GHz machines with thousands of cores, 128 TB of RAM and FTL internet connections, tech companies don't really have any incentive to actually reduce bloat.

Edit: it's still quite sad. I feel like we had languages with a way better future, and more promising programming architectures, back in the 80s.

Post reply on HN