First thing i do is set an alert when costs go over 10$ for any new project. Highly recommend
I got pwned by my cloud costs
31–40 of 657 posts
Re: I got pwned by my cloud costs
#32Ouch. If Troy Hunt of all people can make this mistake, it can happen to anybody. HIBP is an awesome service funded totally by donations, so it's too bad this happened. Of course Microsoft is happy to hide behind their confusing pricing model and let customers overpay for Azure without alerting them.
> If Troy Hunt of all people can make this mistake, it can happen to anybody.
Exactly this. As a low-level / embedded / non-cloud stuff dev, I've been getting up to speed through all the cloud-ification of the industry, but I'm still scared (not literally ofc) of running most things on my own on any big cloud provider (smaller ones seem more manageable).
I'm reading this and seems like being a customer of cloud services is like walking a dangeous path filled with gotchas and caveats, just jumping from cover to cover while hiding from danger, and hoping you're safe and didn't mess it up so far, "fingers crossed".
Like this tiny detail that he didn't realize was critical, so I would fall on it too plus on another 500s small papercuts: "oh I set cache up, so I hope all is well". "Yeah, no you aren't, I guess you didn't think of this detail about maximum cached file size! Gotcha, Game Over!"
Yeah cloud providers should have clearer communitacions and etc etc... but the fact of today is that they don't. So I'd never sleep well feelin 100% confident that I had covered and taken into account every minuscule detail and possible scenario that could end up being a disaster.
Re: I got pwned by my cloud costs
#33Earlier quoted context omitted.
As opposed to all those other customers who are not good faith actors?
You’re trying to be snarky to GP, why exactly? Yes there are bad faith actors that might try to get some free cash out of cloud refunds. And other customers can also be good faith actors and included in the assertion. The post applies to everyone and I’d second it. Ask nicely for a refund in these situations, the worst that can happen is they say no. Where did they say that “only Troy Hunt shall receive a refund, for…
The response was snarky but it's meaning is true. Troy Hunt will get the refund without any problems because he is a public figure. But if a John Doe will make the same mistake he can only hope someone at AWS/Azure/GCP will lift his fees too - which is not guaranteed!
Re: I got pwned by my cloud costs
#34Have you contacted Azure? On one hand you owe the money “fair and square”, but on the other if I were them I’d waive an unexpected $10k bill to a good faith actor that was incurred without any proactive notification by Azure.
OP do this! It works, they are usually very generous (same for gcloud!)
Re: I got pwned by my cloud costs
#35It would be really classy if MS forgave that debt, especially considering the service is a public benefit.
Would be even classier if the major cloud providers responded to customers calling out for budget limits for the past decade. Not many people want to risk potentially infinite costs.
Re: I got pwned by my cloud costs
#36I wonder if before cloud computing, has there ever been a successful product / service where it was accepted with just a shrug that the volatility of monthly costs means it could bankrupt you with next month's bill, because of complexities and opaqueness of the cost structure make it virtually impossible to predict and protect against extreme peaks in all parts of the setup.
Even if you run a relatively opaque cost structure business like a restaurant, you can still calculate the maximum cost of ingredients for one month, the salaries, energy, etc. if you simply use the "best case scenario" of having every seat at every table booked for all opening hours, with people ordering your most sold dishes. Cloud computing is still leagues above that in terms of cost predictability.
I once worked for small, non-startup software company who pondered moving servers to Azure. The Azure partner shop analysed the needs and came up with a monthly cost "between 30k and 120k per month". They were really surprised the company stuck with their non-cloud setup because "everybody is moving into the cloud!!"
Re: I got pwned by my cloud costs
#37Earlier quoted context omitted.
Every cloud makes this mistake easy! You have to manually activate billing alerts for everyone because they want you to spend more snd more each month. I am still waiting for a cloud without these dark patterns. But that will never happen because it‘s leaving a big amount of money on the table by not being hostile.
Get a VPS from linode for $5 a month and it costs $5 a month.
Sure that's what i am doing. A beefy dedicated machine with no bandwidth pricing. But that also means i need to do everything by my own. I don't get any of the worry-free services of AWS.
Re: I got pwned by my cloud costs
#38It is worth mentioning that the alert itself costs money. So if you're evaluating the alert every 5 minutes on the past 24h of data it can burn a small but surprising amount of money. From TFA it looks like that would be 10 cents per "time series". Or what I translate it to, is 10 cents every 5 minutes (*I think, but I havent used Azure in some time*). $1.20/hour, $28.80/day, almost $900/month. Not too hard to drop t…
A time-series represents a "thing you're monitoring" – in this instance, it's aggregate egress, so $0.10 per month, regardless of the evaluation period.
Monitoring CPU? Another $0.10 per month. Memory? Another $0.10.
Thankfully, not $900.
Re: I got pwned by my cloud costs
#39That's the typical story. Something goes wrong and it costs you (typically a small company) a lot of money. At that time just nobody is looking at metrics. Even alarms don't help absolutely because they can also be missed. The only thing that would really help were a hard spending limit that stops all services except storage. If your site is important there will be such an amount of user feedback that it is impossibl…
Alerts can also fail to be timely due to mail/SMS/other delivery issues, or the right people being in the middle of something else. This delay means it is still possible to rack up and unexpected cost.
Or they can fail completely.
And the alerts themselves cost if you want something reliable so you have to weight that against the danger. Pay as you go cloud can be a maze of costing concerns..
> The only thing that would really help were a hard spending limit that stops all services except storage.
Yep. Though that is small comfort if you need to guarantee more than a couple if 9s of uptime, hopefully those with that requirement can soak up the unexpected billing blips.
Re: I got pwned by my cloud costs
#40>What we're talking about here is egress bandwidth for data being sent out of Microsoft's Azure infrastructure (priced at AU$0.014 per GB).
AUD $0.014 is roughly USD $0.01. Which I thought was reasonable. But on [1] only "Data transfer between Availability Zones(Egress and Ingress)" cost $0.01. Do transferring from Azure to CF count as that? Other Internet egress (routed via Routing preference transit ISP network) starts at $0.08
I hope someone from Azure CS could give him a custom discount.
It is also worth thinking, the cost HIBP saved on Cloud / Serverless over the years could have wiped out ( if not more ) by this single incident.
[1] https://azure.microsoft.com/en-au/pricing/details/bandwidth/...