Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

271–280 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#271

Post by BeefWellington »

api wrote: post #214
BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. Adding images, videos, other large attachments, rich search, and all the advertising and billing and analytics stuff would blow this out of…

    I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation.
I agree mostly. Where I differ in that I would argue that hashtags were THE thing that Twitter is most known for but that could be a perspective from having been on the platform for forever and a day and I recognize not everyone may make that same association anymore.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#272

Post by LastTrain »

sitkack wrote: post #180

I am both embarrassed and disappointed with the negativity this post has attracted. A litanny of "gotchas", where someone attempts to best the OP. What about x, y and z? It can't possibly scale. Twitter is so much more than this, etc. The OP isn't making the assertion that Twitter should replace their current system with a single large machine. The whole thread paints a picture of HN like it is full of a bunch of hal…

Yours is by far the most negative post in this thread - in tone and content. I’m sorry but this kind of proposal is intentionally controversial and invites picking through it like this. Glad the author has thicker skin than you.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#273

Post by doublepg23 »

narag wrote: post #260
TacticalCoder wrote: post #244

TFA, to me, touches about something I've wondered about a very long time ago: what are the implications of CPU and storage growing at much faster rates than human population? Back in the 486 days you wouldn't be keeping, in RAM, data about every single human on earth (let's take "every single human on earth" as the maximum number of humans we'll offer our services to with on our hypothetical server). Nowadays keeping…

TFA, to me, touches about something I've wondered about a very long time ago... It was a little more than ten years ago for me. I realized that a hard disk could store a database of every human alive, including basic information, links (family relations) and maybe a photo. I still don't know what the implications are. Maybe we don't want to know, but it's not really that difficult to think about.

Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (ex: I have a co-worker who's name is incorrectly spelled three different ways in prod services) the hard part?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#274

Post by [deleted] »

sitkack wrote: post #180

I am both embarrassed and disappointed with the negativity this post has attracted. A litanny of "gotchas", where someone attempts to best the OP. What about x, y and z? It can't possibly scale. Twitter is so much more than this, etc. The OP isn't making the assertion that Twitter should replace their current system with a single large machine. The whole thread paints a picture of HN like it is full of a bunch of hal…

[deleted]

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#275

Post by TideAd »

qaq wrote: post #201
pixl97 wrote: post #111

Earlier quoted context omitted.

Replication and simple never belong in the same sentence. DNS which is one of the simplest replication systems I know of has its own complex failure modes.

It can be outsourced complexity as in just using horizontally scalable database like Spanner, CockroachDB etc.

CockroachDB is nice to use but every database has complexity you have to deal with.

Here's one I ran into recently: if a range has only 1 of 3 replicas online then it will stop accepting traffic for that range until it has 3 replicas again.

(for the folks at home, "range" is a technical term for 512 bit slice of the data - CRDB replicates at the range level)

So, in some code I wrote, I had account for not only 1) the whole DB being unavailable but also 2) just one replica being unavailable (they're different failure modes that say different things about the health of the system).

It's a good behavior! Good for durability. But I had to do some work to deal with it, spend an hour coming up with a solution, etc. There are databases that work at Twitter scale but no there are no silver bullets among those that do. You need full time engineers to manage the complexity and keep it online, or else it could cost the company shitloads of money - I've seen websites of similar scale where a two-hour outage cost them $20 million.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#277

Post by Hasu »

hinkley wrote: post #232
keewee7 wrote: post #63

In the coming years we will probably see a lot of complicated microservice architectures be replaced by well-designed and optimized Rust (and modern C++) monoliths that use simple replication to scale horizontally.

Yes, but it won't be because they're better, it'll be because they're different. Like last time, and the time before that, and the time before that, and the time before that.

You're right, no progress has ever been made in software, no new ideas are better than any old ideas, and it's fads all the way down. The only difference between software today and software in 1980 is that today's software is hip and software from 1980 is square.

I understand the frustration with flavor of the week "best practices" and the constant churn of frameworks and ideas, but software engineering as a practice IS moving forward. The difficulty is separating the good ideas (CI/CD, for example) from the trends (TDD all the things all the time) ahead of time.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#278

Post by sulam »

trishume wrote: post #69
BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashta…

You’re missing metadata in your size estimates.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#279

Post by BeefWellington »

trishume wrote: post #69
BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashta…

I think the difficult part would be that tagging and indexing the relationship between a single tweet and all of its component hashtags (which you would then likely want metrics on to avoid needing to count indexes on, etc.) is where it would really start to inflate.

Another poster dug into some implementation details that I'm not going to go into. I think you could shoehorn it into an extremely large server alongside the rest of your project but then you're looking at processing overhead and capacity management around the indexes themselves starting to become a more substantial part of processing power. Consider that for each tweet you need to break out what hashtags are in it, create records, update indexes, and many times there's several hashtags in a given tweet.

When I last ran analytics on the firehose data (ca. 2015/16) I saw something like 20% of all tweets had 3 or more hashtags. I only remember this fact because I built a demo around doing that kind of analytics. That may have changed over time obviously, however without that kind of information we don't have a good guesstimate even of what storage and index management there looks like. I'd be curious if the former Twitter engineers you polled worked on the data storage side of things. Coming at it from the other end of things, I've met more than a few application engineers who genuinely have no clue how much work a DBA (or equivalent) does to get things stored and indexed well and responsively.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#280

Post by metadat »

wtallis wrote: post #262
trishume wrote: post #205

Earlier quoted context omitted.

That's really cool! Each year of historical images I estimate at 2.8PB, so it would need to scale quite far to handle multiple years. How would you actually connect all those external drive chassis, is there some kind of chainable SAS or PCIe that can scale arbitrarily far? I consider NVMe-over-fabrics to be cheating and just using multiple machines and calling it one machine, but "one machine" is kinda an arbitrary…

Stuff like [0] exists, allowing you to fan out a single server's PCIe to quite a few PCIe JBOD chassis. Considering that SSDs can get you ~1PB in 1U these days, you can get pretty far while still technically sticking with PCIe connectivity rather than NVMeoF. [0] https://www.liqid.com/products/liqid-elements/liqid-48-port-...

I was skeptical about 1PB in 1U, but searched and learned this was showcased by Supermicro as far back as 2018:

https://www.supermicro.com/en/pressreleases/supermicro-unlea...

Post reply on HN