Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

191–200 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#191

Post by morphle »

tylerhou wrote: post #60
morphle wrote: post #22

Why not a single FPGA with 100Gbps ethernet or pcie with NVM attached? Around $5K for the hardware and $5K for the traffic per month. The software would be a bit trickier to write, but you now get 100x performance for the same price

A bit trickier is a huge understatement.

That depends. If you would use our asynchronous runtime reconfigurable array called Morphle Logic [1] instead of FPGA (Field Programmable Gate Array), you could program this hardware Twitter in a 1000 hours and have it run at 50 Tbps. You would not need to know much about hardware but you would get a thousandfold speedup of your software.

[1] https://github.com/fiberhood/MorphleLogic/blob/main/README_M...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#192

Post by sgt »

wonnage wrote: post #48

This doesn’t seem to support fetching a specific tweet by id?

No need, for those you can display a "Twitter is experiencing issues" or something similar. That will encourage the user to return to the main app and enjoying infinite scroll.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#193

Post by mrtksn »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Twitter's "linked" tweets seems to be strangely unattached from context.

What I mean is, Twitter seems to be processing data based on whatever it is in the tweet and doesn't maintain some grand coherent database.

So I changed my Twitter handle and opened a new account with my original Twitter handle and to my surprise, I was receiving notifications of engagement with tweets my old account sent previously.

I also heard that a method for spamming Twitter trending topics is to send tweets and delete them quickly.

My impression is that Twitter is big on real time processing. They definitely don't search the entire database for #YOLO tweets, instead they seem to be searching the almost-live stuff and some archived stuff(probably ranked and saved as noteworthy tweets or something).

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#194

Post by sayrer »

sayrer wrote: post #161
sayrer wrote: post #137

Earlier quoted context omitted.

"web previews: I'd do this by making it the client's responsibility." Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering ( https://www.jwz.org/blog/2022/11/mastodon-stampede/ ) For blocks/mutes, you have to account for retweets and quotes, it's…

Here are some pointers: "Our approach to blocking links" https://help.twitter.com/en/safety-and-security/phishing-spa... "The Infrastructure Behind Twitter: Scale" https://blog.twitter.com/engineering/en_us/topics/infrastruc... "Mux" https://twitter.github.io/finagle/guide/Protocols.html#mux I do agree that some of this could be done better a decade later (like, using Rust for some things instead of Scala), but it wa…

Here's the Twitter edge server from years ago: https://courses.cs.washington.edu/courses/cse551/15sp/notes/...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#195

Post by morphle »

tluyben2 wrote: post #31
morphle wrote: post #22

Why not a single FPGA with 100Gbps ethernet or pcie with NVM attached? Around $5K for the hardware and $5K for the traffic per month. The software would be a bit trickier to write, but you now get 100x performance for the same price

That would be quite a nice project for fun.

I have seen a number of papers of people doing such a fun project as their PhD thesis. I'll try to find some more [1] examples with links on Google Scholar. Try searching for p4 programmable dataplane content address".

[1] Flexible Content-based Publish/Subscribe over Programmable Data Planes https://pure.rug.nl/ws/files/206093441/Flexible_Content_base...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#196

Post by quickthrower2 »

JosephRedfern wrote: post #149
drewg123 wrote: post #42

How much bandwidth does Twitter use for images and videos? Less than 1.4Tb/s globally? If so, we could probably fit that onto a second machine. We can currently serve over 700Gb/s from a dual-socket Milan based server[1]. I'm still waiting for hardware, but assuming there are no new bottlenecks, that should directly scale up to 1.4Tb/s with Genoa and ConnectX-7, given the IO pathways are all at least twice the bandwi…

I suppose that in practice you'd need to consider burst bandwidth and not just 95/99 percentiles.

… HN thread that reinvents CDN …

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#197

Post by Gigachad »

_zoltan_ wrote: post #178
cortesoft wrote: post #95

Earlier quoted context omitted.

It is way more than 1.4TBs a second globally.

in this specific discussion it's very important to use Tb and TB correctly.

This is never going to happen. If I want to indicate it's correct I'd write Tbit/s or Tbyte/s. Otherwise its a coin flip if TB and Tb has been used correctly.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#198

Post by skywhopper »

koolba wrote: post #154
xyzzy123 wrote: post #140

Earlier quoted context omitted.

I wonder how much is api traffic and how much is assets & images.

I wonder how much of that is crypto spam bots replying to each other.

Can you detect and block those spam bots with less effort than it would take to process them?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#199

Post by jeffbee »

lazyasciiart wrote: post #85
jeffbee wrote: post #12

I like this kind of exercise. One thing I am not seeing is analytics, logs and so forth that as I understand it are significant portions of Twitter's production cost story.

If it's this cheap to run you don't need analytics because you don't need to monetize it, and if it's this simple you don't need logs because it'll all work correctly the first time!

How do you implement trending without logs?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#200

Post by yazzku »

traceroute66 wrote: post #35
andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

> I find it much more appealing to just make the whole thing run on one fast machine. Indeed. Lots of examples out there, one being Let's Encrypt[1] who run off one MySQL server (with a few read replicas but only one write). [1] https://letsencrypt.org/2021/01/21/next-gen-database-servers...

Damn, that was a good and sober read, thanks for sharing.
Post reply on HN