If anything, this shows the insane scalability of the cloud
I got pwned by my cloud costs
581–590 of 657 posts
Re: I got pwned by my cloud costs
#582Ouch. If Troy Hunt of all people can make this mistake, it can happen to anybody. HIBP is an awesome service funded totally by donations, so it's too bad this happened. Of course Microsoft is happy to hide behind their confusing pricing model and let customers overpay for Azure without alerting them.
Correct me if I'm wrong, but Troy Hunt is a person focusing on security, not infrastructure, deployments or development even. If anyone is near making that mistake, it's people like Troy Hunt. Operators would of course see the problem easily (paying for bandwidth like that would be the first warning sign), while they are sometimes blind to other issues, like security.
> Correct me if I'm wrong, but Troy Hunt is a person focusing on security, not infrastructure, deployments or development even.
Eh, I don't know - either way he is a Microsoft Regional Director and MVP for Cloud (as well as security), runs courses on cloud deployment on Pluralsight, and has done speeches on Azure and reducing cloud bills, so if a he can get stung it doesn't say a whole lot good about my chances.
Re: I got pwned by my cloud costs
#583Earlier quoted context omitted.
There's an argument to be made for quality of life for your employees. As someone who has transitioned from on-prem server management to mainly cloud work, my job happiness has skyrocketed. I haven't set foot in a data center in three years and I do not miss it one bit. Dealing with hardware failures, hardware vendors, confusing licensing, having to know SKUs, racking new cabinets, swapping hard drives, patching serv…
I always was a software developer first, but in the old days I spent enough time in the server rooms doing all sorts of sysadmin work, and those days I dabble in devops. And, honestly, I miss the old days. Today, $cloud has some weird spasms where you suddenly get an influx of connection timeouts or tasks waiting for aeons to get scheduled and you just can't log in to a switch or a machine and figure out what the exa…
Cloud popped up mostly due to ease of use. Its a lot easier to hire cloudops engineer with somehow enough knowledge to deploy something on the cloud than someone who will be managing a datacenter and have it running.
The later ppl still do what they did, they just work for Cloud Providers making probably quite a bit more than they did previously.
IMHO its a win win situation for everybody. Less skilled engineers can be “productive” and former sysadmins have huge salaries.
Re: I got pwned by my cloud costs
#584My (naive) solution. Every new account by default has an SMS alert that trips at $100. It says
"Your account has exceed $100 spend. Reply 'SHUTDOWN' to shutdown all services, 'STOP ALERTS' to never see this alert again, or 'DOUBLE TRIGGER' to double the alert trigger value to $200."
$100 is arbitrary, it could be any nominal sum. The idea being that the user can double the alert each time they get it just from SMS. I bet 95% of users would double their alert limit to a comfortable point. The other ~5% will be power users who customize their alerts.
The idea that these companies couldn't know what limits customers want is kinda silly. We can use the same techniques for alerts that we use in algorithms for expanding vector storage, for example. We can "amortize" alerts, so to speak.
Re: I got pwned by my cloud costs
#585Earlier quoted context omitted.
Yeah agreed I haven't worked with Google scale companies but I've always found scaling issue to to development related not infrastructure related. So examples would be a bad db query that takes the system down, overly chatting webserver that issues too many queries to the backend, pulling large datasets into the webapp causing exhaustion of memory ...etc. AWS / Azure can't be these issues they have to be fixed in you…
> I've always found scaling issue to to development related not infrastructure related. So examples would be a bad db query that takes the system down, overly chatting webserver that issues too many queries to the backend This is actually one of the strengths of the cloud, startups that can't afford talent throw compute resources at the problem. Running your own servers isn't hard per se, but it requires a certain br…
Having someone with good cloud chops is still a difficult ask.
Putting it all on the devs is exactly how you end up in the haveibeenpwned database and on the cover of magazines (for the wrong reasons).
We’ve traded sysadmins for more expensive DevOps. I would love to see a study on if we actually hire less people than if we just did it the old school ways.
Re: I got pwned by my cloud costs
#586Earlier quoted context omitted.
IMO this is something which ought to be written into law. It'd be easy to implement a kill switch, and would actually encourage innovation, as people would feel more empowered to experiment with the technology.
Absolutely, and I would make it a bit broader: Anything that automatically charges a client a variable amount should have a maximum-spend limit that the client can set, and it should default to a reasonable number based on the client's expected usage. In fact, you could even just change that to any auto-billing service or product and the default for constant-charge services would simply be the amount of the constant…
Yes. And the law would simply state that the customer may not be billed more than that maximum for the service period. Up to the business to restrict their use of the service, if they want to.
If it's really so 'complicated' that that part can't be done, well, I'm sure it would suddenly become a business priority to cut off use when the limit is hit, if the business can't charge for it.
Re: I got pwned by my cloud costs
#587Earlier quoted context omitted.
Would you mind sharing the services you’ve found that have fixed prices? I haven’t had much luck finding services like that (although I’m looking in the < $20/month range).
For fixed price and fixed performance you can use bare metal providers with unmetered bandwidth generally tier 2 vendors offer that. At $20 bare metal is not easily possible, the lowest prices I have seen are usually 40-50 and above. Howveve you can get a VPS with unmetered bandwidth and no other costs at your price range [1]. The price is still fixed some performance variances may be there, at $20 minor variances ar…
Thanks! I had not heard of OVHcloud before. That looks like what I want.
Re: I got pwned by my cloud costs
#588Earlier quoted context omitted.
This reads like a software engineer being happy work caters lunch so he/she didn’t have to cook for the whole team anymore. Didn’t anyone discuss maybe hiring a cook?
Yes but soon then you're running a kitchen and then a cafe and catering business, as well as a software startup. Which, given how many startups had in-office lunch/food pre-covid is maybe not a bad way to think of that.
Ah yes the Maserati problem https://www.quora.com/Whats-a-Maserati-Problem
By the time you're running a [catering business | massive sysadmin team], you're already a huge success. Congratulations!
Re: I got pwned by my cloud costs
#589Earlier quoted context omitted.
A time-series represents a "thing you're monitoring" – in this instance, it's aggregate egress, so $0.10 per month, regardless of the evaluation period. Monitoring CPU? Another $0.10 per month. Memory? Another $0.10. Thankfully, not $900.
I meant to emphasize frequency, not eval period. Apologies. That said I took a look at the pricing docs and didnt see frequency mentioned, so hopefully I am in the wrong about the price. As an aside, their (Azure's) pricing docs are written in the same fishy way their technical docs are written (my opinion only)...
The service being used here is Azure Monitor, which charges $0.10/month for each metric you monitor with alert rules: https://azure.microsoft.com/en-us/pricing/details/monitor/
They do charge by the MB for ingesting custom metric data, but storage outbound bandwidth is one of the free "standard metrics" built into the platform. So the alert's full cost is $0.10 per month. Since they charge by the "time series," I think that means the OP could configure different alerting thresholds on storage outbound bandwidth, and they'd still only be on the hook for $0.10/month.
Re: I got pwned by my cloud costs
#590Earlier quoted context omitted.
I don't disagree; but I think the cloud (AWS/Azure/GCP) have sort of shielded people from how cheap/powerful the underlying hardware has became. For ~100eur/month on hertzner you can get a 16core Zen3, 128GB RAM with 8TB of NVMe SSD. Unless your stack is horrendously badly optimised you can serve SO MUCH traffic off that - definitely billions of postgres records without breaking a sweat. So the scale argument somewha…
> I don't disagree; but I think the cloud (AWS/Azure/GCP) have sort of shielded people from how cheap/powerful the underlying hardware has became. I guess I shouldn't be surprised, but I do find myself often surprised to realize that for a younger generation of developers they have never experienced hosting on bare metal. So they have not been exposed to costs & benefits vs. the cloud approach and feel that no local…
As a software developer, I think the best thing about the cloud is knowing that if you need the capacity, and it makes sense cost-wise, you'll get it. In-house servers might be cheap, but in my experience it could be incredibly hard to get that money spent when it's needed, and I've seen companies throw expensive software engineering time at optimizing software when it would have been much cheaper to solve the problem with hardware.
Not only can you end up spending $10k of engineering time to optimize and test a random, non-core-competency bit of code instead of an extra $1k/year on hardware, you also have to maintain the optimized code instead of the simpler code.
Maybe I just worked at companies that did a poor job of managing servers, or had a dysfunctional relationship between software engineering and operations, but at least that's no longer something I have to worry about in a cloud environment. If spending a little extra on hardware is the best solution to the problem, process/planning/politics won't get in the way.