Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless: A lesson learned the hard way

sourcebox.be

Post reply on HN
Print view View original

41–50 of 139 posts

Re: Serverless: A lesson learned the hard way

#41

Post by falcolas »

numbsafari wrote: post #4

"The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future." Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. Also, never use such services without being sure to configure such settings. I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial of Capital. Bes…

Amazon has always been happy to DDOS your wallet - one of their whitepapers a few years back on how to survive a DDOS attack was "out scale it".

I can't imagine this changing.

Re: Serverless: A lesson learned the hard way

#42

Post by bluGill »

tdb7893 wrote: post #34
eridius wrote: post #28

Earlier quoted context omitted.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

Could you just do serverless where it starts to rate limit one you reach a certain cost? It seems like this is an issue that could be fixed somehow

Yes, but that isn't always the right answer. If your system is starting to cost "a lot" is that because of a bug (this case), or is it because your idea just "went viral" and you are not getting tons of paying customers signing up?

If it is the latter you do not want any rate limiting, you want everything to scale as fast as possible (I hope there are no bugs on your end). Rate limiting means that your new customers get a poor experience and so they are more likely to ask for a refund, or not renew next time.

Re: Serverless: A lesson learned the hard way

#44

Post by kristianc »

occultist_throw wrote: post #18
mipmap04 wrote: post #15

Earlier quoted context omitted.

Azure has this. When you hit your spending limit, it shuts down your services.

They didn't a year and a half ago. Created a 3k bill for my employer over RemoteApp. Yeah, charge per user, they said. Oh yeah, min 20 users, and we round up - of course in small print at the time. Unless I have hard guarantees, I give "cloud providers" re-loadable cards. Can't take more money than what's on there.

They can terminate your AWS/Azure/DO account though.

Re: Serverless: A lesson learned the hard way

#45

Post by tobyjsullivan »

>This is probably the most stupid thing I ever did. One missing return; ended up costing me $206.

No, dear author. Setting up the AWS billing alarm was the smartest thing you ever did. It probably saved you tens of thousands of dollars (or at least the headache associated with fighting Amazon over the bill).

Developers make mistakes. It's part of the job. It's not unusual or bad in any way. A bad developer is one who denies that fact and fails to prepare for it. A great developer is one like the author.

Re: Serverless: A lesson learned the hard way

#46

Post by ryandrake »

bluGill wrote: post #42
tdb7893 wrote: post #34

Earlier quoted context omitted.

Could you just do serverless where it starts to rate limit one you reach a certain cost? It seems like this is an issue that could be fixed somehow

Yes, but that isn't always the right answer. If your system is starting to cost "a lot" is that because of a bug (this case), or is it because your idea just "went viral" and you are not getting tons of paying customers signing up? If it is the latter you do not want any rate limiting, you want everything to scale as fast as possible (I hope there are no bugs on your end). Rate limiting means that your new customers…

It's almost as if... they should offer multiple options so customers could choose based on their business/hobby needs:

1. Warn me at $X but don't throttle me for any reason--I'll pay if I go viral

2. Warn me at $X and start throttling until I get to $Y at which point stop service and stop charging

3. Warn me at $X and stop service/charging immediately

Re: Serverless: A lesson learned the hard way

#47

Post by ajmurmann »

cddotdotslash wrote: post #17

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

What you say is if course true. However, with EC2 that risk is mostly limited to the code launching EC2 instances. Serverless expands that risk to your application code. There are just way more opportunities to screw up in a way that directly hits your wallet.

Re: Serverless: A lesson learned the hard way

#48

Post by dvfjsdhgfv »

numbsafari wrote: post #4

"The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future." Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. Also, never use such services without being sure to configure such settings. I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial of Capital. Bes…

Amazon refuses to set up bill capping, even though users have been asking for it for many years: https://forums.aws.amazon.com/thread.jspa?threadID=58127&sta...

Re: Serverless: A lesson learned the hard way

#49

Post by rockostrich »

ajmurmann wrote: post #47
cddotdotslash wrote: post #17

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

What you say is if course true. However, with EC2 that risk is mostly limited to the code launching EC2 instances. Serverless expands that risk to your application code. There are just way more opportunities to screw up in a way that directly hits your wallet.

The same would apply to auto-scaling with EC2 instances though. If you were scaling resources based on a queue and made the mistake of adding something to the queue every time you finished something on the queue then you would start to use too many resources.

Without autoscaling, you would just have a queue that grows until the machine runs out of disk space. Either way, this was a problem with code and not event based scaling.

Re: Serverless: A lesson learned the hard way

#50

Post by CupOfJava »

cddotdotslash wrote: post #17

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

Maybe the budget notification was late.
Post reply on HN