Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We handle 80TB and 5M page views a month for under $400

blog.polyhaven.com

Post reply on HN
Print view View original

41–50 of 426 posts

Re: We handle 80TB and 5M page views a month for under $400

#41

Post by gwern »

PragmaticPulp wrote: post #33
mike_d wrote: post #27

Earlier quoted context omitted.

The bulk of the traffic is serving static files. We figured out how to do this with "old and weak" hardware in the 90s. Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59

> The bulk of the traffic is serving static files. The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc. A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it. > Here is a 3.3 GHz Xeon, 24 GB of RAM,…

Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month doing pretty much that, multi-terabyte image datasets and gigabyte-sized DL models. Plus a few websites, which you've probably seen. Zero optimization, just rsync and nginx with defaults, works fine. (It'd be way cheaper except I wanted another 20TB or so of rust to get room to add more datasets to serve - after all, the server is barely breaking a sweat.)

Re: We handle 80TB and 5M page views a month for under $400

#42

Post by NorwegianDude »

mike_d wrote: post #12

It is annoying that the answer is basically "we paid someone [Cloudflare] to do it." In light of that I decided to run some of my own numbers and figure it out. They are pushing about 245 Mbps out of Cloudflare (averaged over the month). Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume. Cloudflare dumps 40% of their traffic over peering, putting their price at $14.70/mo.…

> Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume.

10 cents is far from the lower end for high quality IP transit.

Cloudflare doesn't pay anywhere near $14.70/mo for 80 TB/mo. Even I spend much less than that on 80 TB in a month. On top of that, peering makes it even cheaper for Cloudflare, as you said. Cloudflare is very likely using peering for much more than 40 % of the traffic now, so it's even cheaper for them.

The worst peering region for Cloudflare was North America in 2016 with 40 % peering according to Cloudflare blog posts.

Re: We handle 80TB and 5M page views a month for under $400

#43

Post by PragmaticPulp »

herpderperator wrote: post #40
PragmaticPulp wrote: post #33

Earlier quoted context omitted.

> The bulk of the traffic is serving static files. The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc. A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it. > Here is a 3.3 GHz Xeon, 24 GB of RAM,…

What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM... that completely negates any SSD advantage. SSDs have little to no benefit in serving static files. If you're really strict on this it's trivial to preload these files in RAM after compiling your assets so you don't have to wait for a request.

> What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM

They're running a dynamic website with an API and other dynamic functions.

It's not a static fileserver.

It might make more sense if you visit the website in question: https://polyhaven.com/

Re: We handle 80TB and 5M page views a month for under $400

#44

Post by christophilus »

walrus01 wrote: post #21
mike_d wrote: post #12

It is annoying that the answer is basically "we paid someone [Cloudflare] to do it." In light of that I decided to run some of my own numbers and figure it out. They are pushing about 245 Mbps out of Cloudflare (averaged over the month). Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume. Cloudflare dumps 40% of their traffic over peering, putting their price at $14.70/mo.…

An actually capable dedicated bare metal server with a good amount of RAM, CPU and disk, and 1Gbps will be more like $250-300/mo USD. This would be approximately equivalent in spec to what you could build for $2500 purchase cost if buying a 1U machine and colocating itself. For 30 bucks a month you'll get something very old and weak.

These are pretty capable in my experience: https://www.netcup.eu/vserver/

I’m a happy customer of theirs. The UI for managing things is a disaster, but once you get ssh access, it doesn’t really matter.

Re: We handle 80TB and 5M page views a month for under $400

#45

Post by metadat »

PragmaticPulp wrote: post #31
mike_d wrote: post #12

It is annoying that the answer is basically "we paid someone [Cloudflare] to do it." In light of that I decided to run some of my own numbers and figure it out. They are pushing about 245 Mbps out of Cloudflare (averaged over the month). Wholesale IP transit prices are anywhere from 10 cents to a dollar per Mbps depending on volume. Cloudflare dumps 40% of their traffic over peering, putting their price at $14.70/mo.…

> A capable dedicated server with 2 TB of disk and 1 Gbps unlimited bandwidth will run about $30 in a major European metro, maybe double that for the US. Not even close to comparable. I think you're ignoring all of the functions they're paying for. They're not just hosting a few files. Also, your "worst case" is literally the most optimistic best case for a standalone single server, which completely disregards any re…

But they could deliver potentially ~300TB with a 1gbps unmetered link. Not that hard to come by, could even get to 3PB for $300/mo.

Back in the day we used Texan Colo data centers with DrFTPD to do just this at massive scale.

Bytes in, bytes out. Not a popular opinion here, but I firmly believe it's not necessary to play the CentralizedFlare game to get a winning outcome.

Re: We handle 80TB and 5M page views a month for under $400

#46

Post by PragmaticPulp »

gwern wrote: post #41
PragmaticPulp wrote: post #33

Earlier quoted context omitted.

> The bulk of the traffic is serving static files. The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc. A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it. > Here is a 3.3 GHz Xeon, 24 GB of RAM,…

Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month doing pretty much that, multi-terabyte image datasets and gigabyte-sized DL models. Plus a few websites, which you've probably seen. Zero optimization, just rsync and nginx with defaults, works fine. (It'd be way cheaper except I wanted another…

> I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month

You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain.

The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo.

I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.

Re: We handle 80TB and 5M page views a month for under $400

#48

Post by est31 »

PragmaticPulp wrote: post #33
mike_d wrote: post #27

Earlier quoted context omitted.

The bulk of the traffic is serving static files. We figured out how to do this with "old and weak" hardware in the 90s. Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59

> The bulk of the traffic is serving static files. The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc. A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it. > Here is a 3.3 GHz Xeon, 24 GB of RAM,…

24 GB of RAM means a lot of ability to cache data in RAM, which the OS does already by default. It of course depends on the type of content you have but generally traffic follows a power law, so most content can be served from RAM directly. The write side I'm a bit more worried about, as e.g. SQLite issues a sync request on each transaction (not sure about the other SQL implementations, should be similar there tho), and those might require seeking.

Re: We handle 80TB and 5M page views a month for under $400

#50

Post by PragmaticPulp »

metadat wrote: post #45
PragmaticPulp wrote: post #31

Earlier quoted context omitted.

> A capable dedicated server with 2 TB of disk and 1 Gbps unlimited bandwidth will run about $30 in a major European metro, maybe double that for the US. Not even close to comparable. I think you're ignoring all of the functions they're paying for. They're not just hosting a few files. Also, your "worst case" is literally the most optimistic best case for a standalone single server, which completely disregards any re…

But they could deliver potentially ~300TB with a 1gbps unmetered link. Not that hard to come by, could even get to 3PB for $300/mo. Back in the day we used Texan Colo data centers with DrFTPD to do just this at massive scale. Bytes in, bytes out. Not a popular opinion here, but I firmly believe it's not necessary to play the CentralizedFlare game to get a winning outcome.

> But they could deliver potentially ~300TB with a 1gbps unmetered link. Not that hard to come by, could even get to 3PB for $300/mo.

Cloudflare Pro is $20/month per domain.

Trying to run everything through a single server in a single datacenter just doesn't compare for globally-accessed websites like this, even with a 1Gbps unmetered link.

Post reply on HN