Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

351–360 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#351

Post by Dylan16807 »

threeseed wrote: post #343
Dylan16807 wrote: post #306

Earlier quoted context omitted.

It is not trivial at all. That is a huge amount of scraping. Any machine that can do that will be a similar spec to what you need for serving queries. Not as fast as google does it, but a good amount of them.

It is trivial compared to the task of translating a user query into search results. It's akin to saying the magic behind OpenGPT is the dataset.

Keyword search is not that hard. Pagerank is not that hard.

The comparable goal to the article is to be a search engine, not to fight google for best results.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#352

Post by bithaze »

gorgoiler wrote: post #342
BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Isn’t a hashtag just another kind of user account — an account from which anyone can post? The data structures for the @BeefWellington timeline of tweets and the one for the #BeefWellington timeline of tweets could look roughly the same.

Hashtags aren't like user accounts, no - they're strings that are part of a tweet. In theory, a separate data structure shouldn't be needed since you can just search the full text of tweets, but in practice, I don't know how that scales for the number of all-time tweets.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#353

Post by nimish »

threeseed wrote: post #132
sethev wrote: post #23

John Carmack tweeted something that made me noodle on this too: >It is amusing to consider how much of the world you could serve something like Twitter to from a single beefy server if it really was just shuffling tweet sized buffers to network offload cards. Smart clients instead of web pages could make a very large difference. [1] Very interesting to see the idea worked out in more detail. [1] https://twitter.com/i…

> just shuffling tweet sized buffers to network offload cards Except that's not what it is doing at all. It assembles all the Tweets internally, applies an ML model to produce a finalised response to the user.

Great, staple a few ML accelerators to your NIC. Nvidia sells them! You could build an entire supercomputer style setup 100% optimized for Twitter data movement and computation with COTS hardware IMO.

I strongly doubt that entire datacenters need to be used if and only if Twitter obsessively optimized for hardware usage efficiency over everything else. In reality they don't and make some pretty big compromises to actually get stuff built. Hardware is cheap, people are not.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#354

Post by nimish »

mpoteat wrote: post #58
morphle wrote: post #22

Why not a single FPGA with 100Gbps ethernet or pcie with NVM attached? Around $5K for the hardware and $5K for the traffic per month. The software would be a bit trickier to write, but you now get 100x performance for the same price

Let's spend multi-million dollars a year on a team of highly specialized FPGA engineers writing assembly and HDL so that we can save 5k a month. Feature velocity will be 100x slower as well, but at least our application is efficient. I think that this may make sense for some applications, but I also think that if you can utilize software abstractions to improve developer efficiency, it reduces risk in the long run.

>Let's spend multi-million dollars a year on a team of highly specialized FPGA engineers

I agree with you specialists are expensive but even a team of software engineers runs into the multi million dollar territory.

Why not spend the same amount AND cut down resource use? Hyperscalers have shifted to custom hardware already.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#355

Post by NavinF »

sayrer wrote: post #137
trishume wrote: post #121

Earlier quoted context omitted.

I think I'm pretty careful to say that this is a simplified version of Twitter. Of the features you list: - spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution…

"web previews: I'd do this by making it the client's responsibility." Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering ( https://www.jwz.org/blog/2022/11/mastodon-stampede/ ) For blocks/mutes, you have to account for retweets and quotes, it's…

I'll go ahead and quote that blog post because they block HN users using the referer header.

---

"Federation" now apparently means "DDoS yourself." Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops responding.

The server is basically unusable for 30 to 60 seconds until the stampede of Mastodons slows down.

Presumably each of those IPs is an instance, none of which share any caching infrastructure with each other, and this problem is going to scale with my number of followers (followers' instances).

This system is not a good system.

Update: Blocking the Mastodon user agent is a workaround for the DDoS. "(Mastodon|http\.rb)/". The side effect is that people on Mastodon who see links to my posts no longer get link previews, just the URL.

---

I personally find this absolutely hilarious. Is that blog hosted on a Raspberry Pi or something? "Over a thousand" requests per second shouldn't even show up on the utilization graphs on a modern server. The comments suggest that he's hitting the database for every request instead of caching GET responses, but even with such a weird config a normal machine should be able to do over 10k/second without breaking a sweat.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#356

Post by steviesands »

spullara wrote: post #331

When I was working there I implemented my patent during a hack week (given a set of follows return the list of matching tweet ids, very similar to his prototype): https://patents.google.com/patent/US20120136905A1/en (licensed under Innovators Patent Agreement, https://github.com/twitter/innovators-patent-agreement ) I could have definitely served all the chronological timeline requests on a normal server with lower l…

A few thoughts. The first is, are we asking the wrong questions? Should it be, "If I spend 10m on hardware for predicting ads (storage/compute) that generates 25m in revenue, should I buy the hardware?". Sure, we can "minify" twitter, and it's a wonderful thought experiment, but it seems devoid of the context of revenue generation.

The second is, it's interesting to understand social media industry wide infra cost per user. If you look at FB, Snap, etc. they are within all within an order of magnitude in cost per DAU (DAU / Cost of revenue) of each other. This can be verified via 10-ks which show Twitter at $1.4B vs. SNAP 1.7B Cost of Revenue. The major difference between the platforms is revenue per user, with FB being the notable exception.

Also would you summarize the patent/architecture? The link is a bit opaque/hard to read.

Note: Cost of Revenue does also include TAC and revenue sharing (IIRC) and not just Infra costs but in theory they would also be at similar levels.

eg. SNAPs 10-k https://d18rn0p25nwr6d.cloudfront.net/CIK-0001564408/da8288a...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#357

Post by marcusae313 »

therealdrag0 wrote: post #222
mapme wrote: post #177

Earlier quoted context omitted.

They did not spend half their revenue on compute. It’s more like 20-25% for running data enters/staff for DCs. Check their earnings report. Whats app is not an applicable comparison because messages and videos are stored on the client device. Better to look at Pinterest and snap, which spend a lot on infra as well. The issue is storage, ads, and ML to name a few. For example, from 2015: “ Our Hadoop filesystems host…

Also Search (the article did says these wouldn’t fit to be fair but the discussion seems to be ignoring how much wouldn’t fit and why). Search is pretty expensive especially since to have it responsive you need the indexes to fit in memory—at least the Lucene variety, which at least in old YouTube videos Twitter used.

Twitter still uses Lucene. They built a custom codec for it, but doesn't change the cost too much.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#358

Post by Joeri »

seanhunter wrote: post #336
jameshart wrote: post #311

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory. Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux beca…

The thing is that hardware scales faster than humanity. When the internet boom happened there was no choice except to scale horizontally to reach a global audience, but as this article points out that assumption might no longer hold true for many services. It might make sense to return to vertically scaled highly reliable servers to achieve software simplicity and a lower overall cost.

I’m always reminded of how stackoverflow essentially runs off a single database server. If they can do it, most web properties can do it.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#360

Post by lelandfe »

ninth_ant wrote: post #267
lelandfe wrote: post #174

Earlier quoted context omitted.

If an inspector reviews your house and finds no issues, that is indeed evidence of absence.

This is critically wrong, and misses the point of the cliché entirely. Absence of evidence, in your case via a clean building inspection, does not mean the building is safe. It just means the checklist of known items was considered and nothing bad found. Ask a building inspector if their clean report proves nothing is wrong with the building. They will be firm and quick to inform you that it’s not a warranty — anythi…

I believe you have conflated "proving a negative" with "evidence of absence".
Post reply on HN