Live data from Hacker News

Please fix the AWS free tier before somebody gets hurt

cloudirregular.substack.com

151–160 of 457 posts

Re: Please fix the AWS free tier before somebody gets hurt

#151

Earlier quoted context omitted.

They definitely need a senior executive to stand up and say, "The Customer wants us to be transparent in billing, fix that now." Then they need to start a team dedicated to finding a good way to let customers halt spending at a given limit with minimal impact on their operations. They already win on UX (okay, okay, it's an opinion ffs), but unlimited liability makes a lot of people very uncomfortable. Those two actio…

Have to agree with the others. For example, while setting up ELB it's possible to select at least one option (un-checking the Public IP box) that causes the setup to just fail with a nonsensical error message. Turns out ELB requires a public IP to communicate with the nodes. That's just the most glaring one off the top of my head. I also remember trying to set up SFTP whenever it was first released. It was literally…

How are people not building in Terraform for an easy ‘destroy’ at the end?

I know it’s rhetorical and a lesson learned myself, but yeah… I would expect folks to learn to use this tool to help manage costs this way.

Re: Please fix the AWS free tier before somebody gets hurt

#152
Oracle - of all businesses - got this right. I know, I'm in shock too.

In Oracle Cloud Infrastructure, after your credit is exhausted, you have to explicitly opt in to billing or else they stop all paid services for you.

Moreover, you can choose to keep billing disabled and use their free services without fear of unknown costs.

AWS have just decided to run with a policy of offering refunds when people make mistakes. Unfortunately, some people are ignorant of this, or too timid to ask for their money back.

Re: Please fix the AWS free tier before somebody gets hurt

#153
I want to point out that I just tried to access my AWS Educate Account today to explore the panels and found:

"ALERT-1: Session time behavior and instance types in your Starter Account will be changing on May 11th, 2021. After this date: 1. When your session ends, your resources will be “stopped.” You will be required to re-start your resources when you start a new session. 2. Updates will be made to available instance types. We recommend you to complete currently running work in your Starter Account by May 10th, 2021 as work using instance types that are no longer supported will be lost after that date."

Im still thinking about what can I do with 3 hours (duration of a session) of EC2 computing power...

Re: Please fix the AWS free tier before somebody gets hurt

#154

AWS employs cost obfuscation by design otherwise the default view when you open the console would show you all of your current active services. Not only is that not the case, a single screen to show you all of your current active services doesn't exist. You need to take a deep dive into cost explorer (assuming you have access in corporate land) and try to decipher in what that all means.

If I go to billing I can see an itemized breakdown of every service I'm using. What is wrong with that?

Re: Please fix the AWS free tier before somebody gets hurt

#155

Earlier quoted context omitted.

Have to agree with the others. For example, while setting up ELB it's possible to select at least one option (un-checking the Public IP box) that causes the setup to just fail with a nonsensical error message. Turns out ELB requires a public IP to communicate with the nodes. That's just the most glaring one off the top of my head. I also remember trying to set up SFTP whenever it was first released. It was literally…

cdk is promising for things like this, way safer and easier than rolling your own scripts with bash or python.

Even CDK sucks though because if you’re still kinda new to it all, you want to login and make sure that it’s all hooked together correctly. And you’re back using their shitty UI.

Why you can’t look at the load balancer, look at the listener and then show what’s in the target groups is beyond me.

Re: Please fix the AWS free tier before somebody gets hurt

#156

Earlier quoted context omitted.

IIRC the excuse is that billing is a separate department and they count all the usage dollars way after you done using it, not in realtime. You would still be able to go over your limit and then who should foot the bill? Realtime counting would be too difficult to figure out, our brightest minds are busy figuring out engagement metrics.

This is my personal opinion, though I do work at AWS: It's not that real time counting is difficult, it is that the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical. There is a reason why banks and financial institutions generally do batch processing in the off peak hours when electricity is cheaper and there is less demand for the compute…

I also work at AWS (nowhere near billing), so the usual disclaimers apply, but:

I actually have no idea if billing is real-time or not? I think it's mostly batch, but the records aren't necessarily, though they may be aggregated a bit.

The general point in this thread certainly holds: our systems provide service first, bill second, and that by throwing a record over the wall to some other internal system. It's not unthinkable they could tally up your costs as you go, but the expense has fundamentally already happened, and that's the disconnect.

It would be hard to react ahead of time. Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons. Expensive big-bang on/off things, probably simpler, though I can think of a few sticking points.

It would be hard to react after the fact, too. Where does a bill come from? My own team is deeply internal, several layers removed from anything you're recognize on an invoice, but we're the one generating and measuring costs. Precise attribution would be a problem in and of itself- cutting off service means traversing those layers in reverse, then figuring out what "cut off" means in our specific context. That's new systems and new code all around, repeat for all of AWS- there's a huge organizational problem here on top of the technical one.

I could see some individual teams doing something like this for just their services, but AWS-wide would be a big undertaking.

I wish we had it- I'd sleep a little better at night, myself- but from my limited perspective, it sure looks like we're fundamentally not designed for this.

Re: Please fix the AWS free tier before somebody gets hurt

#157

AWS employs cost obfuscation by design otherwise the default view when you open the console would show you all of your current active services. Not only is that not the case, a single screen to show you all of your current active services doesn't exist. You need to take a deep dive into cost explorer (assuming you have access in corporate land) and try to decipher in what that all means.

Why the f do I have to use a service like billgist to get a breakdown of my services? AWS billing IS complicated but could be made so much simpler...

I agree, having set up serverless and lambda for an api/app that was to be used a few times per day, the billing made no sense at all, it would increase even though the services were used less and was difficult to find what the costs were or whether they could be reduced, eventually I had to shut it down and create a non-lambda solution somewhere else because I had little control of the cost.

Re: Please fix the AWS free tier before somebody gets hurt

#159
The issue here is really Billing... AWS Budgets allows you to set a budget as far down as $1 USD... however because billing is done piecemeal, it is possible to have up to 24 hour delay for charges...

You can spend $500,000 on AWS in 24 hours, it's not difficult, it's not easy but you totally can blow past your "budget" because it's not a hard cap.

Re: Please fix the AWS free tier before somebody gets hurt

#160
This is something I think about a lot. For work, I run a cloud service, and being able to accomodate students (and other cost-sensitive users) like the one in the post is important to me. I've always thought that letting people experiment over the weekend is what leads to people wanting to use something at work (where the real budget is), and I think that supporting those users is how you build a userbase of advocates. Maybe they can't afford (or justify) the enterprise plan quite yet, but they can still be a happy user and cheer on those that can justify the expense.

I've personally found that it can be hard to get the approval to type in your company credit card until you've done enough research to prove that something is going to be worth the money. That leads to a chicken and the egg problem; people won't be confident until they've paid, but they can't gain that confidence until they pay! So you have to get the small "testing" use case perfect, or you'll never have any real customers. (My corollary to this is because AWS already has your company credit card, and you already have "root" or similar because of your role, it sure is easy to build whatever you want there. If AWS provides some service, you can start using it and the cost will be lost in the noise. But if it's not on AWS, then you'll have to produce some justification to use the service before you can start paying for it. I ran into that a lot at my last job; I wanted to buy $5/month services like Sentry but was told no, whereas other people could just create a m4.4xlarge RDS instance for $1000/month and nobody even noticed or cared. People really like being approvers at the time of entering the credit card, rather than at the time of actual cost accrual, and cloud providers really facilitate that. Not sure that's really helping those approvers -- it almost feels a little bit like embezzlement.)

Anyway, ranting aside, here's what we do for the cloud service I work on:

1) We have a free tier. It's really free; you don't even get to enter a credit card. Sign in and start doing your work. (We delete your stuff after 4 hours, though.)

2) For the paid tier, all costs are pay as you go. The instant you click "delete workspace", no more costs accrue. Merely having an account open doesn't accrue any costs for you, and there is no way to create a phantom resource that you can't see in the UI and delete. If you delete everything, billing is over.

One weakness that I'd like to fix is the latency between resource use and when we tell you about it. That takes a few days, so if you are playing with aggressive autoscaling, you don't really know what it's going to cost until your experiment is over. I'd like to collect real-time usage data and just bill based off of that, so that the UI can update you within seconds of your job starting. If it's too much, you can just pull the plug and not be surprised.

The next step is letting people pre-pay, and do what the vast majority of comments on this thread want: kill the compute resources when the budget is exceeded. My thought is that it's hard to ask your customers for money upfront, which is probably where the post-pay model originated. I personally always have reservations about buying 3 year reserved instances from cloud providers, even if it saves a ton of money. "What if we stop using it tomorrow!?" But there is probably a good compromise here: type into the UI what you'd like to pay for autoscaling per month, and once that budget it exceeded, run at the bare minimum "keep the lights on" level. More difficult than the alternative, but certainly possible. And very good for users -- no total outage, no unexpected bill they have no hope of being able to pay. Things are just slower for a while.

Anyway, I don't know what the perfect formula for cloud pricing is -- but it's clear that what AWS has is not quite right, and that we can probably do better. To paraphrase Jeff Bezos ("your margin is my opportunity"), what AWS has consistently done wrong for years is your chance to make it better and get paid for doing so.

Post reply on HN