Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got pwned by my cloud costs

troyhunt.com

Post reply on HN
Print view View original

451–460 of 657 posts

Re: I got pwned by my cloud costs

#451

Post by zo1 »

hardwaresofton wrote: post #180
dx034 wrote: post #154

Earlier quoted context omitted.

How's that? Setting up a revers nginx proxy with cache takes probably less than an hour even if you've never done it (speaking from experience). And otherwise, if the files don't change that much just ssh in, copy them on the server and serve them via nginx and cloudflare tunnel? I'm in no way a sysadmin and have set up these configurations manually in less than an hour for side projects. Cloudflare tunnel also allow…

Ah sorry I should have been clearer on this -- "global cache" === CDN. Hetzner does incur a performance latency (unless you use the brand new US DC of course, and your customer happens to be in the US). IIRC right now you can't mix US cloud servers and German ones in the same Load balancer (also a relatively new hetzner cloud feature) but of course you can do some DNS tricks and get the loads to be fast. Actually hos…

Not related to your comment 100%, but after reading your comment I went researching curiously. Ended up questioning "hey didn't ISPs used to cache content?" Only to discover that they don't anymore, because of HTTPS/SSL, the gift that keeps on giving and effectively warping the web.

So that leads me to my question for HN. Have we completely abandoned non-HTTPS, particularly perhaps for the use-case of server-side caching of HTTP content? Also, isn't this a valid use-case to not use HTTPS and to re-enable that sort of functionality at the network/ISP level?

Re: I got pwned by my cloud costs

#452

Post by KerryJones »

buro9 wrote: post #265
sascha_sl wrote: post #258

Earlier quoted context omitted.

Or simply use a proper CDN that doesn't pretend to eat all the cost for a flat fee but then sometimes does not. BunnyCDN has an amazing volume tier at half a cent per GB.

Oh exactly that. Or if caching is your biggest priority then Fastly or Akamai will shine too. But if you're balancing all considerations and want the cheap "good enough" caching with the DDoS protection, free TLS certs, and unmetered (assuming you aren't imgur or something)... then Cloudflare does a great job at being good enough. And for those sharp edges... drop in a proxy of your own, or layer your CDNs.

In this scenario are you saying

AWS/Azure > BunnyCDN > Cloudflare?

Or just straight AWS/Azure > Cloudflare?

Re: I got pwned by my cloud costs

#453

Post by rawtxapp »

buro9 wrote: post #107

Don't put Cloudflare in front of a Cloud egress bill. i.e. don't do this: Azure|Amazon > Cloudflare Always use your own proxy where the egress is well within your free tier, i.e. do this: Azure|Amazon > Hetzner|Linode > Cloudflare Why? Because Cloudflare cache is a massively multi-tenant LRU cache and whilst hot files will be cached well (and with Cloudflare Tiered Cache even better - but this itself is a cost) anyth…

If you use argo caching on Cloudflare, it should reduce origin server load even more. Essentially, instead of going directly to your origin, cloudflare endpoint will first reach to it's root node to see if it's cached there and only that node is allowed to communicate with your origin. I see like ~95% cache hits with that turned on.

Re: I got pwned by my cloud costs

#454

Post by ajmurmann »

nostrebored wrote: post #380
XCSme wrote: post #321

Earlier quoted context omitted.

> Azure|Amazon > Hetzner|Linode > Cloudflare Why not directly Hetzner|Linode > Cloudflare?

So that you incur as much downtime risk as possible, obviously. I hate these 'cloud economics' optimizations that people tend to try.

The best setup will forever remain Heroku free instance tier with a free Pingdom account providing traffic to keep it from getting shutdown

Re: I got pwned by my cloud costs

#455

Post by StefanKarpinski »

b112 wrote: post #347
windexh8er wrote: post #339

Earlier quoted context omitted.

The simple answer here is that Troy was using Cloudflare to offset costs he knew he would incur with Azure. He states verbatim: "Firstly, I always knew bandwidth on Azure was expensive and I should have been monitoring it better, particularly on the storage account serving the most data." ...and he didn't have simple monitors in place to alert him of uncommon billing spikes. I get your point, if he's not OK with usin…

There are no savings with cloud, weaving or not. You want to waste money? Hire a car, with a driver, when you need it. Want to save money. Learn to drive. You always pay more for outsourcing stuff, a lot more, than doing it yourself. You can buy 1000x the processing power, by buying baremetal. You can get 100,000x more bandwith for cost, when not using the cloud. People think baremetal is hard. It isn't. It does take…

This is such a bad and US-centric example: for anyone who lives in a place where it is easy to get around without a car, hiring a car only when you need it is a no-brainer financially and owning a car is a totally waste of money.

Re: I got pwned by my cloud costs

#456

Post by chasd00 »

j1elo wrote: post #32
cyberCleve wrote: post #3

Ouch. If Troy Hunt of all people can make this mistake, it can happen to anybody. HIBP is an awesome service funded totally by donations, so it's too bad this happened. Of course Microsoft is happy to hide behind their confusing pricing model and let customers overpay for Azure without alerting them.

> If Troy Hunt of all people can make this mistake, it can happen to anybody. Exactly this. As a low-level / embedded / non-cloud stuff dev, I've been getting up to speed through all the cloud-ification of the industry, but I'm still scared (not literally ofc) of running most things on my own on any big cloud provider (smaller ones seem more manageable). I'm reading this and seems like being a customer of cloud servi…

Remember when no-sql came out and everyone was rushing to it because "rdbms don't scale"? I'm beginning to feel the same way towards "cloud" in the Azure or AWS sense. You can go really really far with standard issue VMs from linode or digital ocean and so on. I wonder how many are overpaying for Cloud services so far above and beyond what their actual needs are.

Re: I got pwned by my cloud costs

#457

Post by sudhirj »

jrochkind1 wrote: post #384
sudhirj wrote: post #157

This particular problem basically boils down to "CDN providers don't like caching large files", which is a very common problem. Everything else was configured and setup exactly right to not have a large bill. Most CDN providers have a lot of machines out on the edges of their networks, and it's understandable that they don't stuff these machines with large disks, likely preferring smaller faster SSDs. But this is a v…

Cloudfront isn't much discounted bandwidth out compared to S3 though, is it? I see S3 is initial $0.09/GB, going down to $0.07 after 50TB or $0.05 after 150TB. Cloudfront North America is $0.085 for first 10TB; but $0.110 and up for other regions. going down to $0.060 north america after 100TB, and okay $0.025 after 1PB. (but $0.050 and up in other regions even after 1PB). So okay, Cloudfront gets cheaper egress at l…

The reserved capacity pricing is lower, in a business setting your account manager will usually suggest this pretty quickly if you have a steady and/or increasing Cloudfront bill.

Re: I got pwned by my cloud costs

#458

Post by Spooky23 »

Symbiote wrote: post #435
Spooky23 wrote: post #318

Earlier quoted context omitted.

When you are growing, it’s a no brainer. When you are at steady state it depends. As a case in point, I worked in standing up a critical system in a large enterprise a few years ago. We spent about $12M on compute, storage, networking, etc. At operational state, it was about 40% cheaper than AWS. The problem is, it all sat there for 6-18 months filling up before we fully hit that state. With a cloud provider, you pay…

You're an order of magnitude larger than what I work on, but on our last big project we purchased and installed half in the first year, then the remaining half 18 months later.

Keep in mind that size tends to lower intelligence! ;)

Re: I got pwned by my cloud costs

#459

Post by flutas »

Ostrogodsky wrote: post #430
martinald wrote: post #260

Earlier quoted context omitted.

I don't disagree; but I think the cloud (AWS/Azure/GCP) have sort of shielded people from how cheap/powerful the underlying hardware has became. For ~100eur/month on hertzner you can get a 16core Zen3, 128GB RAM with 8TB of NVMe SSD. Unless your stack is horrendously badly optimised you can serve SO MUCH traffic off that - definitely billions of postgres records without breaking a sweat. So the scale argument somewha…

> For ~100eur/month on hertzner you can get a 16core Zen3, 128GB RAM with 8TB of NVMe SSD. What option is that? The closest I see is the CCX41, but that is 40% more expensive, 140 Eur/month, half the RAM (64 GB) and ~4% of the disk space (360 GB) https://www.hetzner.com/cloud

All I can see is maybe the AX101? It matches all the specs they put down, although the SSD is RAID 1 @ 4TB total.

https://www.hetzner.com/dedicated-rootserver/ax101

Re: I got pwned by my cloud costs

#460

Post by ssully »

fastball wrote: post #22
scrollaway wrote: post #20

Earlier quoted context omitted.

You’re trying to be snarky to GP, why exactly? Yes there are bad faith actors that might try to get some free cash out of cloud refunds. And other customers can also be good faith actors and included in the assertion. The post applies to everyone and I’d second it. Ask nicely for a refund in these situations, the worst that can happen is they say no. Where did they say that “only Troy Hunt shall receive a refund, for…

The point is that if they give refunds to all the good actors they won't make any money.

If you have a long time customer (especially one who brings in as much good publicity as Troy Hunt) and you look at their billing history this spike would be a clear anomaly. Writing off an $8k bill to keep a customer around and happy for years to come is worth more than that bill.
Post reply on HN