Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

341–350 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#341

Post by Nextgrid »

SilverBirch wrote: post #109

I think one of the under-estimated interesting points of twitter as a business is that this is the core. Yes, Twitter is 140 characters, it's got "300m users" which is probably 5m real heavy users. So yes, you could do a lot of "140 characters, a few tweets per person, few million users" on very little hardware. But that's why Twitters a shit business! How much RAM did your advertising network need? Becuase that is w…

If it actually takes you a single machine to run this, you don't really need an advertising network to fund it. Out of 5M users (let alone the theoretical 300M) there will be enough people who'd be happy to pay for verification or an exclusive badge on their profile.

> How are your 1,000 engineers going to push their code to production on one machine?

That might actually be the reason why Twitter barely keeps afloat. 1k engineers for a product that's already built and hasn't fundamentally changed nor evolved in years makes me wonder what business value those engineers actually provide.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#342

Post by gorgoiler »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Isn’t a hashtag just another kind of user account — an account from which anyone can post?

The data structures for the @BeefWellington timeline of tweets and the one for the #BeefWellington timeline of tweets could look roughly the same.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#343

Post by threeseed »

Dylan16807 wrote: post #306
threeseed wrote: post #281

Earlier quoted context omitted.

It is trivial to index the same amount of content as Google as the web is mostly static. The hard part is in being able to translate a search query into a list of pages. And that requires a level of sophistication that far exceeds a laptop.

It is not trivial at all. That is a huge amount of scraping. Any machine that can do that will be a similar spec to what you need for serving queries. Not as fast as google does it, but a good amount of them.

It is trivial compared to the task of translating a user query into search results.

It's akin to saying the magic behind OpenGPT is the dataset.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#344

Post by Nextgrid »

thriftwy wrote: post #138

I remember Stack Overflow running on a single Windows Server box and mocking fellow LAMP developers with their propensity towards having dozens of VMs to same effect. That was some time ago, though.

I believe that's still the case: https://stackexchange.com/performance

9 web servers to serve the entire network. I wish more developers were aware of just how performant modern (non-cloud) hardware is.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#345

Post by p-e-w »

nkellenicki wrote: post #334
p-e-w wrote: post #333

Earlier quoted context omitted.

> You certainly could build server applications that way. But it feels like there's something fundamental to how service businesses operate that pushes away from that kind of hyperoptimized model. That "fundamental" thing is the cultural expectation that SaaS offerings constantly grow in features, rather than in reliability or performance. As your example from the world of video games demonstrates, there is no indust…

You're not taking into account data, you're only talking about features. What about when the data no longer fits on the one machine? Or processing the data exceeds the capacity of the machine? Data growth through user growth or just normal day-to-day usage is expected.

If Twitter's data can fit on one machine, then the data of 99.99% of companies can. Not every product needs a billion users with Gigabytes of storage each. The assumption that if your startup's tech isn't scalable enough to become the next Google then it's the wrong tech is hilarious nonsense driven largely by ego fantasies.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#346

Post by Nextgrid »

kevingadd wrote: post #175
lazyasciiart wrote: post #85

Earlier quoted context omitted.

If it's this cheap to run you don't need analytics because you don't need to monetize it, and if it's this simple you don't need logs because it'll all work correctly the first time!

"You don't need to monetize it" who's going to fund your Twitter-as-a-charity? What happens when the free money goes away? Businesses have to pay the bills eventually one way or another, you need to plan for that in advance

If you're a small team running this and can actually deliver it with a single machine (or two), just charging a few bucks a month for verification should net enough money to run it and provide a decent living (and out of 300M users, there will be people who would pay).

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#347

Post by mapme »

p-e-w wrote: post #345
nkellenicki wrote: post #334

Earlier quoted context omitted.

You're not taking into account data, you're only talking about features. What about when the data no longer fits on the one machine? Or processing the data exceeds the capacity of the machine? Data growth through user growth or just normal day-to-day usage is expected.

If Twitter's data can fit on one machine, then the data of 99.99% of companies can. Not every product needs a billion users with Gigabytes of storage each. The assumption that if your startup's tech isn't scalable enough to become the next Google then it's the wrong tech is hilarious nonsense driven largely by ego fantasies.

Twitters data cannot fit on one machine. In 2015 their Hadoop cluster was 30 PB per earlier comments/their blog. How do you fit that on one machine?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#348

Post by Nextgrid »

jupp0r wrote: post #84
fbdab103 wrote: post #59

Earlier quoted context omitted.

I think you should always plan for failures, but modern enterprise hardware is quite reliable. I would even posit that if you stood up a brand new physical server today, it has a good chance of beating AWS uptime (well, not the AWS dashboard numbers) over a one year period.

"hardware is quite reliable" is not a valid strategy. Hardware fails with some non-zero probability. You need to have a plan in place what to do if that happens, taking into account service disruption, backups etc. Having a system in place that handles most of this gracefully (like kubernetes) is one way of having such a plan, there are others. Which one works best is dependent on your app, cost of downtime, your tea…

Keep in mind that the complexity of your distributed system is a liability itself. Most cloud or large company downtimes aren't due to hardware failure per-se but either operator error or unexpected interactions between components of their distributed system. You might just be trading one source of downtime for another.

> the complexities of operating a service well.

Keep in mind that in a lot of business applications, downtime isn't the end of the world and might be an accepted and priced-in "cost" of doing business. Operating it as you consider "well" would just cost them more with no benefit.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#349

Post by nkellenicki »

p-e-w wrote: post #345
nkellenicki wrote: post #334

Earlier quoted context omitted.

You're not taking into account data, you're only talking about features. What about when the data no longer fits on the one machine? Or processing the data exceeds the capacity of the machine? Data growth through user growth or just normal day-to-day usage is expected.

If Twitter's data can fit on one machine, then the data of 99.99% of companies can. Not every product needs a billion users with Gigabytes of storage each. The assumption that if your startup's tech isn't scalable enough to become the next Google then it's the wrong tech is hilarious nonsense driven largely by ego fantasies.

It does not fit on one machine. Tweets alone generate petabytes of data a year, and other events are petabytes per day.

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

https://ankush-chavan.medium.com/twitter-data-storage-and-pr...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#350

Post by NavinF »

trishume wrote: post #205
jiggawatts wrote: post #156

Earlier quoted context omitted.

> which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. You would be surprised . First off, SSDs are denser than hard drives now if you're willing to spend $$$. Second, "plug in" doesn't necessarily mean "in the chassis". You can expand storage with external disk arrays in all sorts of ways. Everything from external PCI-e cages to SAS disk arrays, fibre channel, NVMe-o…

That's really cool! Each year of historical images I estimate at 2.8PB, so it would need to scale quite far to handle multiple years. How would you actually connect all those external drive chassis, is there some kind of chainable SAS or PCIe that can scale arbitrarily far? I consider NVMe-over-fabrics to be cheating and just using multiple machines and calling it one machine, but "one machine" is kinda an arbitrary…

> I consider NVMe-over-fabrics to be cheating

Is an infiniband switch connected to a bunch of machines that expose NVMe targets really that different from a SAS expander connected to a bunch of JBOD enclosures? Only difference is that the former can scale beyond 256 drives per controller and fill an entire data center. You're still doing all the compute on one machine so I think it still counts.

Post reply on HN