Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless: A lesson learned the hard way

sourcebox.be

Post reply on HN
Print view View original

81–90 of 139 posts

Re: Serverless: A lesson learned the hard way

#81

Post by codedokode »

chrisco255 wrote: post #35
eridius wrote: post #28

Earlier quoted context omitted.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

The developer should be writing unit tests for their code so they can avoid small mistakes like this.

It is impractical to cover every line of code with tests (it would get too expensive). Futhermore, in this case the author would have to test production config interacting with Amazon servers rather than a piece of code.

And even 100% code coverage doesn't find all possible errors.

Re: Serverless: A lesson learned the hard way

#82

Post by sudhirj »

dvfjsdhgfv wrote: post #57
benologist wrote: post #24

There's a lesson in there about how AWS makes a crisp $10m dollar bill for the richest man in the world every day.

I know you're being sarcastic but I feel it's partly true. They announce so many different services, each month something new appears, but this very basic feature - bill capping - asked by users from the very beginning, has never been implemented. It's hard to believe they lack the skill or that it would be much more complicated than the current alert system.

According to the folks I've spoken to, they don't do bill capping because they have no way to safely shut down your workloads in any way - they'd prefer to let you know and have you do it. And having that choice is way better than a capping operation destroying your production database or causing downtime for your users.

Re: Serverless: A lesson learned the hard way

#84

Post by sah2ed »

chrisco255 wrote: post #77
chrisco255 wrote: post #35

Earlier quoted context omitted.

The developer should be writing unit tests for their code so they can avoid small mistakes like this.

Down vote for advocating for unit tests? That's just good practice in general.

I think integration tests would be more appropriate here, especially since there are different co-operating moving parts: S3 EC2/Lambda.

Re: Serverless: A lesson learned the hard way

#85

Post by sah2ed »

autotune wrote: post #14
numbsafari wrote: post #4

"The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future." Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. Also, never use such services without being sure to configure such settings. I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial of Capital. Bes…

If you really wanted, you could create a script that after a certain billed amount gets reached switches your site via route53 over to a static s3 page that says "down for maintenance" or something until you figure out why your forecasted billing amount is so high. forcastedSpend is an object you can call via api: http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/... and a SDK like boto3: http://boto3.readt…

EDIT: I've reposted my question as a top-level comment to give it a bit more visibility as I'm interested in seeing answers as to what would be a fair price for such a service.

https://news.ycombinator.com/item?id=14984330

Re: Serverless: A lesson learned the hard way

#86

Post by sudhirj »

You did well to have billing alerts enabled. Exactly the same thing happened to be, but I didn't notice for three months - no emails because I'd created an account and domain for a side project. Didn't notice anything on my card because the charge had been declined, but my bank didn't contact me. Finally found out because I knew the local AWS rep (was the relationship manager for the accounts we use at work). Had to apologise and explain the situation in detail to AWS and they forgave the bill. That was tens of thousands of dollars.

Re: Serverless: A lesson learned the hard way

#87

Post by Piskvorrr »

codedokode wrote: post #81
chrisco255 wrote: post #35

Earlier quoted context omitted.

The developer should be writing unit tests for their code so they can avoid small mistakes like this.

It is impractical to cover every line of code with tests (it would get too expensive). Futhermore, in this case the author would have to test production config interacting with Amazon servers rather than a piece of code. And even 100% code coverage doesn't find all possible errors.

Do you need to cover every _line_ of code, or do you need to test resulting behavior? Also, while nothing is 100% foolproof, the example here would probably have been caught.

Re: Serverless: A lesson learned the hard way

#88

Post by sudhirj »

sudhirj wrote: post #86

You did well to have billing alerts enabled. Exactly the same thing happened to be, but I didn't notice for three months - no emails because I'd created an account and domain for a side project. Didn't notice anything on my card because the charge had been declined, but my bank didn't contact me. Finally found out because I knew the local AWS rep (was the relationship manager for the accounts we use at work). Had to…

For those talking about this being a 'serverless' problem or not, think they point is that it's a lot easier to shoot yourself in the foot. Great power + responsibility, etc. On regular servers (outside of unbounded autoscaling) mistakes cost a flat rate.

Re: Serverless: A lesson learned the hard way

#89

Post by Piskvorrr »

falcolas wrote: post #41
numbsafari wrote: post #4

"The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future." Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. Also, never use such services without being sure to configure such settings. I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial of Capital. Bes…

Amazon has always been happy to DDOS your wallet - one of their whitepapers a few years back on how to survive a DDOS attack was "out scale it". I can't imagine this changing.

OTOH, for anything that's not a toy project, this is one of the viable approaches. Power tool is powerful (which also means you can hurt yourself using it), news at 11.

Re: Serverless: A lesson learned the hard way

#90

Post by sah2ed »

Charged at a flat rate like the cost of a Digital Ocean $5/mo instance, would developers pay for such a service to provide automated notifications of service overages for all the major cloud providers?

All a developer needs to do immediately after adding a credit card to AWS/Azure/GCP would be to create an IAM role with permission to automatically add and track fine-grained billing alarms and notify via email/sms for any potential billing overages.

I think a $60/yr service like this would be useful to protect against future events of bill shock.

Post reply on HN