Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless: A lesson learned the hard way

sourcebox.be

Post reply on HN
Print view View original

131–139 of 139 posts

Re: Serverless: A lesson learned the hard way

#131

Post by lifeisstillgood »

tobyjsullivan wrote: post #45

>This is probably the most stupid thing I ever did. One missing return; ended up costing me $206. No, dear author. Setting up the AWS billing alarm was the smartest thing you ever did. It probably saved you tens of thousands of dollars (or at least the headache associated with fighting Amazon over the bill). Developers make mistakes. It's part of the job. It's not unusual or bad in any way. A bad developer is one who…

Just wanted to say that last paragraph is one of the simplest descriptions of professionalism in the job I have read - accept and prepare for your own human failings :-)

Re: Serverless: A lesson learned the hard way

#132

Post by SimonPStevens »

vacri wrote: post #129
SimonPStevens wrote: post #127

Earlier quoted context omitted.

Yeah, I totally accept that people who want caps may not be in the target audience. That's probably true. But don't pretend it's because the problem is too hard for them to build, which is what was originally claimed. If their big customers wanted it it would get built. Besides, Azure proves that it's clearly possible. Azure have a cap. MSDN gives you free Azure credits. When you open an account via MSDN you still ha…

> S3 data..? Delete it. O_o > ... Make it read only. The primary use-case of s3 is reading objects. This would not be a deterrent for quite a few use-cases > ... Pretend to delete it, but make recovery possible for x days. Still consumes the space that they're charging for in the first place > People who want a cap are going to more concerned with the overspend than any data or service integrity. This is patently not…

I think we are talking cross purposes here. I get what you are saying. A cap that junks your services would certainly not be suitable for everyone, I get that.

But this thread was initially about a dev running some experiments and getting a $200 unexpected bill. They would have been very happy with a $30 cap that just deleted everything. That functionality would be easy to build if they wanted to. But they don't. For other reasons, not because it's hard.

The thing I dont buy is Amazon claiming its too hard to build a cap. A cap that suits some people would be easy. What they really mean is... A simple hard cap is only useful to customers we dont care about because they dont pay us enough. An advanced cap with all the kinds of failover options and thresholds that a medium sized business might want is complex and the people who actually pay the big money (those we care about) don't actually want caps anyway.

Re: Serverless: A lesson learned the hard way

#133

Post by dvfjsdhgfv »

vacri wrote: post #129
SimonPStevens wrote: post #127

Earlier quoted context omitted.

Yeah, I totally accept that people who want caps may not be in the target audience. That's probably true. But don't pretend it's because the problem is too hard for them to build, which is what was originally claimed. If their big customers wanted it it would get built. Besides, Azure proves that it's clearly possible. Azure have a cap. MSDN gives you free Azure credits. When you open an account via MSDN you still ha…

> S3 data..? Delete it. O_o > ... Make it read only. The primary use-case of s3 is reading objects. This would not be a deterrent for quite a few use-cases > ... Pretend to delete it, but make recovery possible for x days. Still consumes the space that they're charging for in the first place > People who want a cap are going to more concerned with the overspend than any data or service integrity. This is patently not…

> The protection you want for hobbyists would destroy small businesses

Are you serious? This would mean Azure is not fit for business:

https://docs.microsoft.com/en-us/azure/billing/billing-spend...

When your usage results in charges that exhaust the monthly amounts included in your offer, the services that you deployed are disabled for the rest of that billing month. For example, Cloud Services that you deployed are removed from production and your Azure virtual machines are stopped and de-allocated. To prevent your services from being disabled, you can choose to remove your spending limit. When your services are disabled, the data in your storage accounts and databases are available in a read-only manner for administrators. At the beginning of the next billing month, if your offer includes credits over multiple months, your subscription will be re-enabled. Then you can redeploy your Cloud Services and have full access to your storage accounts and databases.

The decision not to implement this in AWS has nothing to do with technical issues - they can all be solved in this way or another.

Re: Serverless: A lesson learned the hard way

#134

Post by dvfjsdhgfv »

SimonPStevens wrote: post #132
vacri wrote: post #129

Earlier quoted context omitted.

> S3 data..? Delete it. O_o > ... Make it read only. The primary use-case of s3 is reading objects. This would not be a deterrent for quite a few use-cases > ... Pretend to delete it, but make recovery possible for x days. Still consumes the space that they're charging for in the first place > People who want a cap are going to more concerned with the overspend than any data or service integrity. This is patently not…

I think we are talking cross purposes here. I get what you are saying. A cap that junks your services would certainly not be suitable for everyone, I get that. But this thread was initially about a dev running some experiments and getting a $200 unexpected bill. They would have been very happy with a $30 cap that just deleted everything. That functionality would be easy to build if they wanted to. But they don't. For…

> A simple hard cap is only useful to customers we dont care about because they dont pay us enough. An advanced cap with all the kinds of failover options and thresholds that a medium sized business might want is complex and the people who actually pay the big money (those we care about) don't actually want caps anyway.

That's an honest answer, I'd be happy if they formulated it this way. Fortunately, there are other cloud providers with billing cap implemented properly, and you don't hear horror stories about them (problems with spending too much on AWS are very common though).

Re: Serverless: A lesson learned the hard way

#135

Post by jbergens »

jo909 wrote: post #130
jbergens wrote: post #126

Earlier quoted context omitted.

this was a problem involving multiple parts, unit tests normally don't catches this. You need an integration/functional test and that can be much more time consuming to write for all "integrations" and code paths.

It was a single function that changed behavior after a refactoring. It did work where it did not need to, because the work was already done on the object. This is only hard if you don't test at all and can't already mock the object download/upload or don't have pure functions.

In this case it might have helped, I didn't read the code, but in a more general case these kind of things are rarely found by unit tests. I still doubt that the triggering caused by the file change would have been found in a unit test.

Re: Serverless: A lesson learned the hard way

#136

Post by manigandham »

numbsafari wrote: post #4

"The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future." Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. Also, never use such services without being sure to configure such settings. I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial of Capital. Bes…

This always comes up but people seem to just ignore the complexity: What exactly is supposed to happen when the spend hits your budget?

Even a few bytes sitting on S3 continue to incur charges and it's hard to be real-time with spend tracking at the scale of these providers so the only option they have is to delete your entire account immediately. Is that what you want? Who would?

For most companies, business continuity matters. The proper solution is to use the budget and reporting features to check your work.

Re: Serverless: A lesson learned the hard way

#137

Post by tapirl »

The cost by using AWS is hard to foresee. For years, my s3 storage got charged nothing. But some a month, it got charged several dollars.

I have migrated all my services to GCE. At least GCE provides free decent quotas for every resource.

Re: Serverless: A lesson learned the hard way

#139

Post by cutler »

tyingq wrote: post #124
traduz wrote: post #115

Earlier quoted context omitted.

Yes, but I guess in that case you would put your lambda function behind an API gateway, and limit the user requests. If it's a static content you would serve it from a CDN. Not a specialist on this, but that's what I would do.

Wouldn't an API gateway typically limit requests per IP/end user? I guess it could limit global request rate. But the idea of unbounded elastic services behind a global rate limiter is just funny to me. Like a Ferrari with a 50mph limiter.

Yes, I still don't get it.
Post reply on HN