Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless Horrors

serverlesshorrors.com

Post reply on HN
Print view View original

241–250 of 503 posts

Re: Serverless Horrors

#241

Post by coredog64 »

herpdyderp wrote: post #207
acoustics wrote: post #188

The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and contingent. Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding tho…

The solution is simple: budget caps.

Yes and no. 100% accurate billing is not available in realtime, so it's entirely possible that you have reached and exceeded your cap by the time it has been detected.

Having said that, within AWS there are the concepts of "budget" and "budget action" whereby you can modify an IAM role to deny costly actions. When I was doing AWS consulting, I had a customer who was concerned about Bedrock costs, and it was trivial to set this up with Terraform. The biggest PITA is that it takes like 48-72 hours for all the prerequisites to be available (cost data, cost allocation tags, and an actual budget each can take 24 hours)

Re: Serverless Horrors

#242

Post by McGlockenshire »

stressback wrote: post #121

I read a lot of the posts at the little blog here and, uh, every single one sounds like a complete amateur making a cloud configuration mistake. I haven't found one that is the provider's fault or the fault of "serverless" I would be embarrassed to put my name on these posts admitting I can't handle my configs while blaming everyone but myself. Serverless isn't a horror, serverlesshorrors poster. You are the horror.…

> every single one sounds like a complete amateur making a cloud configuration mistake

Golly if only the configuration wasn't made this way on purpose exactly to cause this exact problem.

Re: Serverless Horrors

#243

Post by general1465 »

I tried AWS serverless, figured out that it is impossible to test anything locally while you are forced to use AWS IAM role for serverless run which has access to everything.

That's just a problem waiting to happen while you are always running tests on production...

Re: Serverless Horrors

#244

Post by coredog64 »

appreciatorBus wrote: post #102
JackSlateur wrote: post #93

Earlier quoted context omitted.

They have billing limits https://docs.aws.amazon.com/cost-management/latest/userguide...

These aren’t limits though, they are just budget notifications. What would be helpful, would be if when you set up your account there was a default limit – as in an actual limit, where all projects stop working once you go over it - of some sane amount like $5 or $50 or even $500. I have a handful of toy projects on AWS and Google cloud. On both I have budgets set up at $1 and $10, with notifications at 10% 50% and 9…

You can attach an action to that budget overage that applies a "Deny" to an IAM and limits costly actions (that's for small accounts not in an Org. Accounts with an Org attached also have the option of applying an SCP which can be more restrictive than an IAM "Deny")

Re: Serverless Horrors

#245

Post by McGlockenshire »

qcnguy wrote: post #91
JJMcJ wrote: post #88

Earlier quoted context omitted.

That's why I like VPS setups. You hit the monthly maximum, and it just stops working. I host demos, not running a business, so it's less of an issue to get interrupted. Better an interruption than a $50,000 bill for forgetting to shut off a test database from last Wednesday.

That's not comparable. With a VPS there is no monthly maximum, just a max load on a second by second basis. You can be hit with traffic of which 90% bounces because your server is down, get nowhere near your intended monthly maximum, and then the rest of the month is quiet.

You seem to be describing this as a bad thing instead of the objectively good thing that it is.

Re: Serverless Horrors

#246

Post by bapak »

VectorLock wrote: post #144
phoenixhaber wrote: post #43

When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony). Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had the…

Amazon refunded you and you hate them for it? I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.

You wake up to a bill of one hundred thousand dollars and now it's up to you to dispute it. I think hating them for it is very fair.

Re: Serverless Horrors

#247

Post by McGlockenshire »

bc569a80a344f9c wrote: post #56
EGreg wrote: post #28

Are there any protections these days at the cloud provider level? Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?

Not _really_. AWS has a budget tool, but it doesn’t natively support shutting down services. Of course, you can ingest the alerts it sends any way you want, including feeding them into pipelines that disable services. There’s plenty of blueprints you can copy for this. More seriously - and this is a legitimate technical limitation - of course AWS doesn’t check each S3 request or Lambda invocation against your budget,…

> but it doesn’t natively support shutting down services [...] of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes

So, in other words, the vendor has provided substandard tooling with the explicit intent of forcing you to spend more money.

Re: Serverless Horrors

#248

Post by bapak »

mahirsaid wrote: post #230

Seem likes there are mistakes that were made on behalf of the users. The attackers found these mistakes and took advantage of them. i don't think "severless" is the problem.

Serverless is the problem in that most serverless services don't let you hard-cap spend.

This issue is serverless-specific. If I pay $20/month on VPN the most frightening thing that can happen is the client calling you about your website being down, not a $100k bill.

Re: Serverless Horrors

#249

Post by the8472 »

falcor84 wrote: post #137
bee_rider wrote: post #122

Earlier quoted context omitted.

I’ve accidentally hit myself with a bigger than expected AWS bill (just $500 but as a student I didn’t really want to spend that much). So I get being annoyed with the pricing model. But, I don’t think the idea of just stopping charging works. For example, I had some of their machine image thingies (AMI) on my account. They charged me less than a dollar a month, totally reasonable. The only reasonable interpretation…

> The only reasonable interpretation of “emergency stop on all charges completely” would be to delete those images I disagree; a reasonable but customer-friendly interpretation would be to move these into a read-only "recycle bin" storage for e.g. a month, and only afterwards delete them if you don't provide additional budget.

Which they already do for account closure, supposedly, I have never tested it.

https://docs.aws.amazon.com/accounts/latest/reference/manage...

Re: Serverless Horrors

#250

Post by estimator7292 »

pixl97 wrote: post #225
rr808 wrote: post #198

Earlier quoted context omitted.

Maybe, but its a huge reason to use real servers instead of serverless.

I mean real servers get hit with things like bandwidth fees so it's not a 100% solution.

Not even remotely the same scale of problem. Like at all.

If your business suddenly starts generating Tbs of traffic (that is not a ddos), you'd be thrilled to pay overage fees because your business just took off.

You don't usually get $10k bandwidth fees because your misconfigured service consumes too much CPU.

And besides that, for most of these cases, a small business can host on-prem with zero bandwidth fees of any type, ever. If you can get by with a gigabit uplink, you have nothing to worry about. And if you're at the scale where AWS overages are a real problem, you almost certainly don't need more than you can get with a surplus server and a regular business grade fiber link.

This is very much not an all-or-nothing situation. There is a vast segment of industry that absolutely does not need anything more than a server in a closet wired to the internet connection your office already has. My last job paid $100/mo for an AWS instance to host a GitLab server for a team of 20. We could have gotten by with a junk laptop shoved in a corner and got the exact same performance and experience. It once borked itself after an update and railed the CPU for a week, which cost us a bunch of money. Would never have been an issue on-prem. Even if we got DDoSed or somehow stuck saturating the uplink, our added cost would be zero. Hell, the building was even solar powered, so we wouldn't have even paid for the extra 40W of power or the air conditioning.

Post reply on HN