I got pwned by my cloud costs • Hacker News Board

Re: I got pwned by my cloud costs

#601

Post by Nextgrid » Tue, Jan 25, 2022, 1:07 AM UTC

dx034 wrote: post #127

alfiedotwtf wrote: post #117

Earlier quoted context omitted.

> The only thing that would really help were a hard spending limit that stops all services except storage. Sadly, I haven't found a way to do that with AWS

It's funny that even Hetzner can do that and AWS can't. Shows that there's no interest from AWS to prevent these things from happening.

To be fair, Hetzner doesn't nickel & dime you on everything.

Some things are legitimately very difficult to meter in a real-time way with adequate performance. Imagine if the S3 server had to do a DB query to lookup your account balance & limit on every HTTP request. That'll completely kill performance and availability (what if the billing DB is down? Should that DB now be replicated across all regions? etc).

The reason a lot of cloud alerts and metrics lag is because billing is done asynchronously by parsing logs way after the actual usage occurred. Of course, the real solution here is to just not bill for things you can't easily measure & limit especially when those things are super expensive, but cloud providers' C-suites have to eat & pay their bills somehow.

Re: I got pwned by my cloud costs

#602

Post by folmar » Tue, Jan 25, 2022, 1:07 AM UTC

dkarl wrote: post #590

jjav wrote: post #554

Earlier quoted context omitted.

> I don't disagree; but I think the cloud (AWS/Azure/GCP) have sort of shielded people from how cheap/powerful the underlying hardware has became. I guess I shouldn't be surprised, but I do find myself often surprised to realize that for a younger generation of developers they have never experienced hosting on bare metal. So they have not been exposed to costs & benefits vs. the cloud approach and feel that no local…

As a software developer, I think the best thing about the cloud is knowing that if you need the capacity, and it makes sense cost-wise, you'll get it. In-house servers might be cheap, but in my experience it could be incredibly hard to get that money spent when it's needed, and I've seen companies throw expensive software engineering time at optimizing software when it would have been much cheaper to solve the proble…

> the best thing about the cloud is knowing that if you need the capacity, and it makes sense cost-wise, you'll get it

It stopped to be the case in pandemic times at most cloud operators due to general hardware and capacity shortage.

Re: I got pwned by my cloud costs

#603

Post by Nextgrid » Tue, Jan 25, 2022, 1:08 AM UTC

Seattle3503 wrote: post #584

My (naive) solution. Every new account by default has an SMS alert that trips at $100. It says "Your account has exceed $100 spend. Reply 'SHUTDOWN' to shutdown all services, 'STOP ALERTS' to never see this alert again, or 'DOUBLE TRIGGER' to double the alert trigger value to $200." $100 is arbitrary, it could be any nominal sum. The idea being that the user can double the alert each time they get it just from SMS. I…

The problem is that metering these services at such granularity is difficult: https://news.ycombinator.com/item?id=30066538

Re: I got pwned by my cloud costs

#604

Post by Nextgrid » Tue, Jan 25, 2022, 1:10 AM UTC

scanr wrote: post #75

I wonder how much of the cloud provider revenue comes from situations like this. I suspect quite a lot. I think that the cloud provider business model that allows for uncapped maximum costs is a bit of a commercial dark pattern. What makes it somewhat more nefarious is that it is relatively easy to blame the customer. I’m not surprised that the cloud providers are quick to refund users as it’s likely that they only d…

> I suspect quite a lot.

I don't actually believe so. Cloud providers are known to refund bills incurred by mistake. They make so much margins on legitimate usage by big companies & startups that it's just not worth burning developer goodwill & potentially waste efforts trying to collect a bill the customer legitimately can't pay (and will guarantee he will never use nor advocate for your service again).

Re: I got pwned by my cloud costs

#605

Post by folmar » Tue, Jan 25, 2022, 1:11 AM UTC

nightpool wrote: post #415

dijit wrote: post #269

Earlier quoted context omitted.

Now they have 9. They still serve a lot more traffic than I do and I have hundreds of instances; thousands of containers.

You have thousands of containers? Physician, heal thyself.

Thousands is not a lot. If you do microservices and have 100 of them, 3 replicas of each for dev, qa and prod, you already are at 900.

Re: I got pwned by my cloud costs

#606

Post by Nextgrid » Tue, Jan 25, 2022, 1:12 AM UTC

kidsil wrote: post #358

Shameless plug - the core of my work is about ensuring these unexpected costs never happen. We have some recent case studies where we've successfully reduced cloud costs by 95% https://www.cloudexpat.com/case-studies/ hi(at)cloudexpat.com - happy to help!

Out of curiosity, do you merely optimize existing cloud usage or do you help your clients move to hybrid/bare-metal?

Re: I got pwned by my cloud costs

#607

Post by Nextgrid » Tue, Jan 25, 2022, 1:15 AM UTC

intricatedetail wrote: post #259

If you are not a VC backed corporation you must be insane to run anything on a "cloud". Why not rent a dedicated server from OVH or others where you can actually control costs and pay 10-100 times less?

Because experience getting shit done using boring tools doesn't translate well to a future career in a VC-backend company wrangling Terraform & YAML files.

Re: I got pwned by my cloud costs

#608

Post by Nextgrid » Tue, Jan 25, 2022, 1:19 AM UTC

laurent92 wrote: post #427

abakker wrote: post #406

Earlier quoted context omitted.

No, no. The costs cloud saves are in staffing and opportunity costs. Everyone knows that it is more expensive than a comparable server, but...it is easy, standard, and available. If you want to a) not have real estate capex, b) not worry about the core ops part of your applications, and c) used to outsource infrastructure to a managed service provider anyway, then Cloud is a viable value prop. Plus, the more of the s…

And better credibility when you say “Our vulnerability was on AWS and configuration is hard, but at least we had the default VPC config” rather than “We maintained our own stack and being sysadmin is hard, and the port was exposed on the web.”

Modern cloud services such as S3 or let's say MongoDB seem to have a lot more security footguns than old-school bare-metal. An S3 bucket misconfiguration exposes your data to everyone even if there was never a reason for that data to ever be exposed to the outside world. On a bare-metal, chuck it in a directory outside your web root and someone will have to actually breach the server before they can steal the data.

Re: I got pwned by my cloud costs

#609

Post by Nextgrid » Tue, Jan 25, 2022, 1:21 AM UTC

phkahler wrote: post #441

b112 wrote: post #347

Earlier quoted context omitted.

There are no savings with cloud, weaving or not. You want to waste money? Hire a car, with a driver, when you need it. Want to save money. Learn to drive. You always pay more for outsourcing stuff, a lot more, than doing it yourself. You can buy 1000x the processing power, by buying baremetal. You can get 100,000x more bandwith for cost, when not using the cloud. People think baremetal is hard. It isn't. It does take…

It's the "weaving" part that has non-specific cost. If you have skill at weaving together pieces of the cloud in an optimal way, you can save money. Just like if you have skill in putting together your own infrastructure you can save money. I can see spending money on services, but I don't understand why people invest brain capacity on vendor-specific solutions.

How much extra time & development effort does it take to weave the services the right way to realize those savings, as opposed to doing it the simpler, old-school way on bare-metal?

In my bare-metal-hosted projects I can afford to do a lot of things that would be a major no-no in the cloud because I have so much hardware resources I can save development time doing things inefficiently and still come out ahead in terms of costs.

Re: I got pwned by my cloud costs

#610

Post by RKearney » Tue, Jan 25, 2022, 1:23 AM UTC

mappu wrote: post #531

klohto wrote: post #494

Earlier quoted context omitted.

You are not allowed to use Lightsail once you use more professional services on AWS atleast per ToS

Do you have a more detailed citation for that? At $DAYJOB we seem to be using Lightsail (for non-cache purposes) along with some "real AWS" resources without a problem,

AWS Service Terms[0]

51. Amazon Lightsail

51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services (e.g., proxying network traffic from Services to the public internet or other destinations or excessive data processing through load balancing or content delivery network (CDN) Services as described in the Documentation), and if you do, we may throttle or suspend your data services or suspend your account.

[0] https://aws.amazon.com/service-terms/