Why cloud bandwidth is so obscenely expensive and what you can do about that • Hacker News Board

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#41

Post by Syonyk » Thu, Oct 26, 2023, 5:20 PM UTC

jauntywundrkind wrote: post #33

Syonyk wrote: post #20

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers,…

If you want bandwidth, vps are quite cheap. 100Mbps averages to a a little over 32TB/mo. There there's tons of low CPU low ram systems with 32TB/mo transfer for cheap. Now under $10/mo: https://lowendbox.com/blog/hosteons-they-survived-dedipath-a... People should rack their own rpi5 or nucs, at least. There's various providers for both. And consider moving up from there. The remaining folks who will accept & rack a m…

Sure, I'll use a $5/mo DO Droplet (maybe $6/mo now for the new ones?) and their 1TB transfer pool if I just need raw bandwidth, but, tbh, not a lot of stuff is just pure bandwidth needs. And "capable cloud boxes for various purposes" start adding up. If I just wanted to shovel out a static website, sure. But I can do that a variety of other ways too.

Once you start running "real" things, CPU and RAM use start climbing - mostly RAM in my experience. I run a Matrix homeserver - Synapse really struggle with can do it in less, and I've done it, but a lot of problems go away when you can just throw RAM at the problem.

Also, most of the cheap VPS boxes are badly IOPS limited on disk - unless you start paying a good bit more for faster disk. I split my box between a 2TB NVMe mirror for VM root drives and "high write traffic" stuff (the gameserver drives and the like), and my large, slow, spinning rust array for backup storage and "big stuff." Again, you can do it with cloud, but storage adds up in a hurry - 24TB of "slow" disk is a grand a month.

It just depends on what you need. I really like cloud boxes for light use "toy" boxes, and if that's what you want, they're great. I just cannot wrap my head around the merits of spending a lot of money for "serious cloud compute" when I can build and rack a similarly capable box for 2-3 months of the cloud spend on a similar system. I understand opex vs capex issues for business, but, man, it sure seems stupid to me.

Then again, I'm the kind of person who's had a box racked up a colo (or somewhere similar - random basements on a good connection are nice) for the majority of my adult life. So I'm pretty clearly biased. But I did do a few years of "Hey, cloud, this is cool" - and eventually sat down, evaluated my needs and the costs, and went back to a colo'd box.

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#42

Post by KronisLV » Thu, Oct 26, 2023, 5:23 PM UTC

adventured wrote: post #23

CodesInChaos wrote: post #12

I think egress would benefit from some regulation. This would enable other companies to compete with specific AWS services, without competing with AWS as a whole. Though I probably wouldn't price cap egress itself, but rather force amazon to offer cheap peering.

Amazon & Co. will always rule for the exact same reason Microsoft Office has/will. It's the combination of services effect creating a natural moat. There's nothing you can do about that other than to break up the large entities, it forms by default unless you do that. It requires a comically enormous burden placed on the entity to break it because of how fantastically strong the combination effect is for customers, c…

> So long as they don't put too many barriers up (raising the cost through unnecessary regulation to cut out the competition, which would benefit AWS), we'll always have competitive alternatives, including running your own infrastructure inexpensively with companies like Hetzner or Cloudflare (depending on what you need).

I mean, if the article mentions small businesses in particular, their needs are probably met by software running on a few VPSes. As long as they don't want to leverage any advanced features that the PaaS/SaaS providers would give them, most of the affordable providers should be enough: Hetzner, Contabo, Time4VPS, or even Scaleway, DigitalOcean, Vultr and so on.

Probably pick a company that has been around for >5 or >10 years instead of something on LowEndBox that might disappear a year or two down the line, but other than that it's likely to be fine (as long as you have backups figured out and have either some redundancy/failover, or just are okay with downtime).

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#43

Post by Syonyk » Thu, Oct 26, 2023, 5:24 PM UTC

washadjeffmad wrote: post #38

Syonyk wrote: post #20

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers,…

> My personal box ... for a bit over $100/mo. Is that just $50/U?

It's $75 for a 1U slot (which somehow gives me 2U of space) with dual power, and the rest is bandwidth and an IP block (I've got a /29 so I can put different services on different IPs, because that's... just a thing I do - I'm fundamentally a "first decade of the 2000s Linux sysadmin" in how I think).

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#44

Post by mcherm » Thu, Oct 26, 2023, 5:24 PM UTC

tinrab wrote: post #39

Is there a way to protect against the "Denial of wallets" attack?

Yes, there is a way to protect against that attack!

All you need to do is have triggers in place that will shut down or slow down your service when the costs are exceeding some amount.

(If you get a moment of viral growth, you can always disable the trigger.)

Unfortunately, AWS makes it extremely difficult to build such a trigger, and I'm not sure about other cloud providers.

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#45

Post by onlyrealcuzzo » Thu, Oct 26, 2023, 5:24 PM UTC

zeroCalories wrote: post #9

onlyrealcuzzo wrote: post #6

TL;DR - author claims the hardest part of running a cloud PaaS is networking, thus the cloud providers charge you a ton for egress to outsource the optimization problem to you. It's not clear to me this claim is true - they're in the business of solving exactly this optimization problem - and they have, they're just capturing maximum profit. If it was really the hardest / most costly - then other providers wouldn't b…

??? This is obviously true. The bandwidth available in a geographic location is, practically speaking, limited. Roads are cheap to pave, but if there was congestion 24/7 and you paved all you could, then you would need to start rationing road usage. For a business the best way to ration is to charge more. Solving this problem requires a massive amount of engineering work, which would also justify the increased cost.

And you're not physically limited by how many machines you can put in a data center?

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#46

Post by Syonyk » Thu, Oct 26, 2023, 5:27 PM UTC

mknocker wrote: post #37

Syonyk wrote: post #20

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers,…

I am thinking about doing something similar. Out of curiosity, do you use a reverse proxy web server to protect from malicious attacks ? That`s probably the only part I would think of leaving in the cloud if I were to host my own servers at home.

What's different about a box hanging out in the cloud with open ports, vs at home or in a colo with open ports?

I use CloudFlare to front most of my web stuff for bandwidth diversion - my blog is a purely static Jekyll site, and I've told CF to "cache everything," so if I get hit with a spike of traffic, it isn't hitting my server (much - comments are still locally hosted with a Discourse forum integration, but if that's overloaded it doesn't block page render).

My home bandwidth simply isn't up to hosting. But if I can ever get someone to tap the fiber a quarter mile from my place and give me a drop, I'd move to hosting at home.

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#47

Post by Syonyk » Thu, Oct 26, 2023, 5:29 PM UTC

swatcoder wrote: post #35

Syonyk wrote: post #20

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers,…

Heathen! Not everyone can afford that PhD in enterprise class hardware management you must have earned to pull this off. And besides, what are you going to do when you wake up tomorrow and you're the next FAANG? You'll never scale to 2B DAU on that box and we all need to be ready for that, just in case. I knew a guy who knew a guy who founded a FAANG, so it could happen to you too. You have to be ready! /s

Oof. Thanks for the /s, I've quite literally heard that argument from people before for "cloud." That you have to design to scale, physical hardware can't scale rapidly, etc.

I assume, if I suddenly had reason to scale, I could call my colo and get another slot from them pretty darn quick. They're local and quite responsive, though I do get a feeling their "public shared rack" offering was created to deal with me. I was the first server in it...

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#48

Post by swatcoder » Thu, Oct 26, 2023, 5:30 PM UTC

proteal wrote: post #34

One thing to keep in mind - if you accidentally run up a bill because of a mistake, there’s a good chance you can reach out to support and they will credit your account. $5k may be a lot to an individual, but the cloud provider’s costs for that service are significantly less and they shouldn’t mind forgiving the charge. Case in point, when the author deleted his account and switched providers, the old host didn’t cha…

That's been true during recent flush years, where money was cheap and nobody really cared, but there are signs all over the place that the attitude is changing. Everything that used to be free is being monetized and everything that was monetized is seeing increased prices.

We should expect that these sort of implicit affordances are being trimmed as well, or at least that they're likely to be soon.

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#49

Post by roncesvalles » Thu, Oct 26, 2023, 5:30 PM UTC

swatcoder wrote: post #35

Syonyk wrote: post #20

What you can do about it is go back to the early 2000s and rack up your own boxes in a colo, with some reasonable unmetered bandwidth allocations, and skip the whole "cloud" nonsense. My personal box is a 6C/12T Xeon from a few years back, 128GB ECC RAM, and about 24TB usable disk plus 2TB NVMe, for running a lot of other VMs that I run actual services in (remote backup, my web presence, communications, game servers,…

Heathen! Not everyone can afford that PhD in enterprise class hardware management you must have earned to pull this off. And besides, what are you going to do when you wake up tomorrow and you're the next FAANG? You'll never scale to 2B DAU on that box and we all need to be ready for that, just in case. I knew a guy who knew a guy who founded a FAANG, so it could happen to you too. You have to be ready! /s

But seriously, for scaling up, just have a plan for how to move things to the cloud quickly. Containerization and building everything on Kubernetes help.

Re: Why cloud bandwidth is so obscenely expensive and what you can do about that

#50

Post by zeroCalories » Thu, Oct 26, 2023, 5:31 PM UTC

toast0 wrote: post #32

zeroCalories wrote: post #9

Earlier quoted context omitted.

??? This is obviously true. The bandwidth available in a geographic location is, practically speaking, limited. Roads are cheap to pave, but if there was congestion 24/7 and you paved all you could, then you would need to start rationing road usage. For a business the best way to ration is to charge more. Solving this problem requires a massive amount of engineering work, which would also justify the increased cost.

Bandwidth in a geographic location is certainly limited... there's only so many fiber strands available, and fiber transceivers only go up to a certain speed, +/- wave division; there's costs of transceivers and what not. But in most places where you'll find datacenters, the limit is rather high. It's also usually possible to run more fiber. If you take a look at the Seattle Internet Exchange [1] which is pretty publ…

There is a difference between being able to run more fiber, and being able to do it cheaply. The more you scale, the more you need everything you depend on to scale. For many places that calculation makes it not worth it.