Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

231–240 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#231

Post by KingLancelot »

trishume wrote: post #16
Cyph0n wrote: post #10

Earlier quoted context omitted.

My bad - not English, but ASCII. The assumed max tweet size is in bytes rather than (UTF) characters.

I specifically assumed a max tweet size based on the maximum number of UTF-8 bytes a tweet can contain (560), with a link to an analysis of that, and discussion of how you could optimize for the common case of tweets that contain way fewer UTF-8 bytes than that. Everything in my post assumes unicode.

UTF-8 is 1-4 bytes per codepoint dude, not 1-2.

280 * 4 = 1120, not 560.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#232

Post by hinkley »

keewee7 wrote: post #63

In the coming years we will probably see a lot of complicated microservice architectures be replaced by well-designed and optimized Rust (and modern C++) monoliths that use simple replication to scale horizontally.

Yes, but it won't be because they're better, it'll be because they're different.

Like last time, and the time before that, and the time before that, and the time before that.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#233

Post by jrockway »

Gigachad wrote: post #197
_zoltan_ wrote: post #178

Earlier quoted context omitted.

in this specific discussion it's very important to use Tb and TB correctly.

This is never going to happen. If I want to indicate it's correct I'd write Tbit/s or Tbyte/s. Otherwise its a coin flip if TB and Tb has been used correctly.

I like /s for bytes and ps for bits. 100Mbps = 100 million bits per second. 100MB/s = 100 million bytes per second. (The capitalization is important of course. I tried writing some examples where it's not preserved and it's too weird. I tend to not use capital letters on things like Slack, but for bytes, you just have to. The difference between milli and Mega can also be important, but since nobody talks about the negative powers with bandwidth, you are probably OK if your shift key breaks.)

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#234

Post by nickstinemates »

racingmars wrote: post #210
Xeoncross wrote: post #188

Earlier quoted context omitted.

I must admit, as a bystander I'm torn here. I love that Tristan put out this post and made it so detailed with plenty of assumptions to cover. I also like to hear about possible issues and assumptions which the crowd calls out. Even naysayers can be helpful. I want both, but I don't want to crowd to go to far and kill the desire to produce this kind of content.

> I want both, but I don't want to crowd to go to far and kill the desire to produce this kind of content. I think it's easy to have both. It's all about the tone of the responses. For example, instead of "your assumptions are wrong, this would collapse because X" or "this is dumb because real Twitter does Y which yours doesn't handle," I think responses could be framed as: "Wow, neat thought experiment! If I were to…

Being nice takes up a lot of space. Straight and to the point without the judgment would be a good compromise.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#235

Post by [deleted] »

therealdrag0 wrote: post #215
mgaunard wrote: post #203

All web and cloud technologies are inherently inefficient, and most programmers don't know networking or even how hardware works sufficiently well to optimize for high througput and low-latency. There was an article just yesterday about how Jane Street had developed an internal exchange way faster than any actual exchange by building it from the ground up, thinking about how the hardware works and how agents can inte…

It’s “easy” to optimize for speed when you build from the ground up with no real customers or feature requirements that you can’t just conveniently ignore.

[deleted]

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#237

Post by vidarh »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Biggest problem with this is the lack of considering analytics.twitter.com and ads.twitter.com. Twitter stores event data about everything that happens to a tweet, and lets you target ads with a lot of precision.

While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet views.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#239

Post by yjftsjthsd-h »

pdpi wrote: post #227
lelandfe wrote: post #208

Earlier quoted context omitted.

Sure, but if someone accuses your house of having issues, and you retort that you've had it inspected by professionals, a reply of "Hah! That's evidence , not proof!" is just a bit smarmy.

A few weeks ago there was in incident[0] in Jersey, where some people called fire fighters one evening because they could smell gas, the fire fighters didn’t find any leaks, and the building literally blew up the next morning. Experts make mistakes, and failing to understand that evidence != proof can literally kill people. Sometimes, making the distinction is smarmy; other times, it’s just being sensible. 0. https:/…

Okay, but... we're spit balling database sizes. None of this is safety critical, or even in the general neighborhood of things where it's important enough to go and mathematically prove that our numbers are perfect.
Post reply on HN