Endless loops are now billing issues, just like a DOS.
Serverless: A lesson learned the hard way
121–130 of 139 posts
Re: Serverless: A lesson learned the hard way
#122>Keep an eye on your logs, test everything again and again.
This is the takeaway quote from this for me.
Re: Serverless: A lesson learned the hard way
#123Earlier quoted context omitted.
It's "serverless" in the sense that if the developer had provisioned a "server" then the max incursion of cost would equal the cost of that server, no more no less. So yeah, let's blame the developer, but let's not play like mistakes don't happen and they're not costly in the "serverless" world.
Use EC2, set an auto scaling threshold on CPU utilization, do something dumb, you’ll find you ran “a server” x n. It’s easy to burn tens or hundreds of thousands ‘accidentally’ on “server”, easier than on serverless. If you’re spending real money, you should have an account team. Talk to them if such a problem happens.
> If you’re spending real money, you should have an account team. Talk to them if such a problem happens
The colloquialism for "real money", at least to me, is "a substantial sum". If that's what you intended, wouldn't it make sense that you wouldn't have an account team if the only time you spent real money was by accident?
Re: Serverless: A lesson learned the hard way
#124Earlier quoted context omitted.
Isn't there also a third danger with anything that scales your bill as your app scales - the possibility of some black hat ddos-ing you for the hell of it?
Yes, but I guess in that case you would put your lambda function behind an API gateway, and limit the user requests. If it's a static content you would serve it from a CDN. Not a specialist on this, but that's what I would do.
Wouldn't an API gateway typically limit requests per IP/end user?
I guess it could limit global request rate. But the idea of unbounded elastic services behind a global rate limiter is just funny to me. Like a Ferrari with a 50mph limiter.
Re: Serverless: A lesson learned the hard way
#125I was about to make the same mistake yesterday , but I had written a validation function that would check inside a folder only, and fortunately I did not upload the file in that folder. And the next morning , I read this article...
Man I gotta be careful LOL
Re: Serverless: A lesson learned the hard way
#126Earlier quoted context omitted.
It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?
The developer should be writing unit tests for their code so they can avoid small mistakes like this.
this was a problem involving multiple parts, unit tests normally don't catches this. You need an integration/functional test and that can be much more time consuming to write for all "integrations" and code paths.
Re: Serverless: A lesson learned the hard way
#127Earlier quoted context omitted.
I'm sorry, I just don't buy that. It doesn't have to be a hard cap, it could be a soft one. i.e. at £x your servers start shutting down, you'll get billed for a few extra minutes over your cap, before things have finished shutting down. Servers are totally capable of being shutdown without destroying databases. Besides, we aren't really talking about production databases at large companies. The people who want caps a…
> The people who want caps are devs learning and experimenting. Then AWS is not focused on their use-case. You can make the argument that they're throwing away potential business here, but AWS is already the gorilla in the room, and people clamour for their products already. A couple of years ago they were rated as being bigger than the next 17 VPS providers combined. > I'm sorry, I just don't buy that. It doesn't ha…
Yeah, I totally accept that people who want caps may not be in the target audience. That's probably true. But don't pretend it's because the problem is too hard for them to build, which is what was originally claimed. If their big customers wanted it it would get built.
Besides, Azure proves that it's clearly possible. Azure have a cap. MSDN gives you free Azure credits. When you open an account via MSDN you still have to put in payment details, but you have the option to enable a hard cap that prevents you spending past your free credits. So Azure have clearly got a solution for stopping all the services when the credit limit is reached.
All of what you describe as problems are just decisions to be made. S3 data..? Delete it. Make it read only. Pretend to delete it, but make recovery possible for x days. Doesn't really matter, just pick one when you build it, and document what it does. People who want a cap are going to more concerned with the overspend than any data or service integrity. They could stick up a disclaimer... "If you enable this cap you data may be destroyed or corrupted if your spending reaches the cap". There are solutions to the implementation problem.
Besides, they probably already have all this code in place. If your payment methods gets declined I'm prepared to bet Amazon don't just let all your services continue running indefinitely because shutting them down automatically is too hard of a problem for them to solve. So any cap could be implemented by just triggering the payment declined function.
Re: Serverless: A lesson learned the hard way
#128Earlier quoted context omitted.
It is impractical to cover every line of code with tests (it would get too expensive). Futhermore, in this case the author would have to test production config interacting with Amazon servers rather than a piece of code. And even 100% code coverage doesn't find all possible errors.
Do you need to cover every _line_ of code, or do you need to test resulting behavior? Also, while nothing is 100% foolproof, the example here would probably have been caught.
I doubt this, a unit test wouldn't have covered the infinite triggering of the created events
Re: Serverless: A lesson learned the hard way
#129Earlier quoted context omitted.
> The people who want caps are devs learning and experimenting. Then AWS is not focused on their use-case. You can make the argument that they're throwing away potential business here, but AWS is already the gorilla in the room, and people clamour for their products already. A couple of years ago they were rated as being bigger than the next 17 VPS providers combined. > I'm sorry, I just don't buy that. It doesn't ha…
Yeah, I totally accept that people who want caps may not be in the target audience. That's probably true. But don't pretend it's because the problem is too hard for them to build, which is what was originally claimed. If their big customers wanted it it would get built. Besides, Azure proves that it's clearly possible. Azure have a cap. MSDN gives you free Azure credits. When you open an account via MSDN you still ha…
> S3 data..? Delete it.
O_o
> ... Make it read only.
The primary use-case of s3 is reading objects. This would not be a deterrent for quite a few use-cases
> ... Pretend to delete it, but make recovery possible for x days.
Still consumes the space that they're charging for in the first place
> People who want a cap are going to more concerned with the overspend than any data or service integrity.
This is patently not true, and is why I think you don't really grok why implementing a cap is difficult. It's specifically why I said "everyone would demand different behaviour at the cap". Some would want only this or that service to stop, for example.
A small business sets up a payment cap and hits that cap because they went viral? BAM, all their block storage, destroyed. All their backups, their analytics, their RDS databases, just gone. Right at the time they needed it most.That's a much harder lesson to deal with than "oops, our bill's a bit high because we made a mistake, can you please forgive it?". Or even "ouch, okay we'll pay it". The protection you want for hobbyists would destroy small businesses that may not understand what is actually meant when that hard cap is hit. It's not that caps aren't doable at all, it's just that they're a wicked problem, and the more you look at it, the more issues you can see.
As for soft caps, what is the functional difference between a soft cap and the billing warnings they already have?
Also, their claim on s3 is "we don't lose objects". Destroying objects because of billing would utterly undermine that claim.
> If your payment methods gets declined I'm prepared to bet Amazon don't just let all your services continue running indefinitely because shutting them down automatically is too hard of a problem for them to solve.
AWS does not destroy your services because of late payment. Source: we've just been in late payment.
Re: Serverless: A lesson learned the hard way
#130Earlier quoted context omitted.
The developer should be writing unit tests for their code so they can avoid small mistakes like this.
this was a problem involving multiple parts, unit tests normally don't catches this. You need an integration/functional test and that can be much more time consuming to write for all "integrations" and code paths.
It was a single function that changed behavior after a refactoring. It did work where it did not need to, because the work was already done on the object. This is only hard if you don't test at all and can't already mock the object download/upload or don't have pure functions.