Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless: A lesson learned the hard way

sourcebox.be

Post reply on HN
Print view View original

91–100 of 139 posts

Re: Serverless: A lesson learned the hard way

#91

Post by Piskvorrr »

godot wrote: post #70
Dolores12 wrote: post #3

"a $180 actual cost. I was left with a light headed feeling, it's a lot of money for me" people still play with fire. limit your losses, go with digital ocean or something for 5$/mo flat no matter what.

His previous blog post actually said that he moved away from the exact $5/mo digital ocean plan you're talking about, to this. I think the author meant to do this less of a "play with fire" way but more of experimenting with new tech way. But yes, I agree that for personal sites, running with your own money, you probably want to stick with something safer like the $5/mo digital ocean box.

"Oh, look, new tech. It shines! It heats! Ouch, it also burns!" The comparison to fire is quite fitting.

Re: Serverless: A lesson learned the hard way

#92

Post by rovr138 »

eridius wrote: post #28
cddotdotslash wrote: post #17

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

> It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money.

We dynamically create and instantiate new servers based on load and if it's sustained for a while. Once it's up, it's added to the load balancer. Once the load of them goes down, it's spin down after it's spent some time idle (it costs to instantiate so might as well keep outside of the queue for a bit before completely removing it).

This all runs automatically. If we don't limit it, it's on us.

How is this not a problem with how he managed it?

> This is probably the most stupid thing I ever did. One missing return; ended up costing me $206.

He clearly mentioned it's his error there.

Re: Serverless: A lesson learned the hard way

#93

Post by 21 »

sudhirj wrote: post #88
sudhirj wrote: post #86

You did well to have billing alerts enabled. Exactly the same thing happened to be, but I didn't notice for three months - no emails because I'd created an account and domain for a side project. Didn't notice anything on my card because the charge had been declined, but my bank didn't contact me. Finally found out because I knew the local AWS rep (was the relationship manager for the accounts we use at work). Had to…

For those talking about this being a 'serverless' problem or not, think they point is that it's a lot easier to shoot yourself in the foot. Great power + responsibility, etc. On regular servers (outside of unbounded autoscaling) mistakes cost a flat rate.

Not necessarily.

With a regular server you could go viral, your server dies, so you lose also without a bound in lost business/good will/whatever.

Also need to take into account the time/effort spent on making the regular server scale, albeit this is also a relatively flat rate.

Re: Serverless: A lesson learned the hard way

#94

Post by 21 »

sah2ed wrote: post #90

Charged at a flat rate like the cost of a Digital Ocean $5/mo instance, would developers pay for such a service to provide automated notifications of service overages for all the major cloud providers? All a developer needs to do immediately after adding a credit card to AWS/Azure/GCP would be to create an IAM role with permission to automatically add and track fine-grained billing alarms and notify via email/sms for…

Azure has a feature on their trial account that when you hit your free limit, you can either:

a) go into credit (so they will charge you at end of month)

b) disable services

Maybe AWS/Google also support a hard limit on spending.

Re: Serverless: A lesson learned the hard way

#95

Post by j45 »

The relatively low barrier to learn just a tiny bit of following a Linode or DO vps hardening & stack setup guide to get an ubuntu server going can go a very long way for development and prototyping environments.

It's gotten much, much easier, and is just another form of command line management, similar to the CLI framework tools with your preferred stack.

Once that first setup is done, similar to setting up a serverless environment, you are generally restoring backups of your base image and beginning projects from there.

It also immensely helps to learn about how to build something to scale that isn't completely reliant on the PaaS layer.

Re: Serverless: A lesson learned the hard way

#96

Post by autotune »

sah2ed wrote: post #90

Charged at a flat rate like the cost of a Digital Ocean $5/mo instance, would developers pay for such a service to provide automated notifications of service overages for all the major cloud providers? All a developer needs to do immediately after adding a credit card to AWS/Azure/GCP would be to create an IAM role with permission to automatically add and track fine-grained billing alarms and notify via email/sms for…

Looks like it's already been done, at least for AWS, based on a few Google results (not that I've used/tested any of these):

https://github.com/Teevity/ice https://billgist.com/ http://cloudcheckr.com/

Re: Serverless: A lesson learned the hard way

#97

Post by nickthemagicman »

eridius wrote: post #28
cddotdotslash wrote: post #17

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

I notice AWS doesn't have any ability to set limits....

Re: Serverless: A lesson learned the hard way

#98

Post by chrisco255 »

chrisco255 wrote: post #35
eridius wrote: post #28

Earlier quoted context omitted.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

The developer should be writing unit tests for their code so they can avoid small mistakes like this.

> because of a refactor, I forgot the return statement and it just continued overwriting the file again

Unit tests are specifically useful for refactors. You can refactor your code and ensure that it behaves as intended. Integration tests are great, too, don't get me wrong. Either or both would have probably caught this.

Re: Serverless: A lesson learned the hard way

#99

Post by emeraldd »

cddotdotslash wrote: post #17

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

You could argue that it's a pay-per-use problem because of inherent unpredictability in that pricing model. When you have "big" resources behind you, it's less impactful but still and issue. The difference with a server based vs. serverless here is how the cost grows. You can predict what the cost of a server is going to be and strictly control it. Can you actually do that with the serverless option?

Re: Serverless: A lesson learned the hard way

#100

Post by cutler »

eridius wrote: post #78
cddotdotslash wrote: post #76

Earlier quoted context omitted.

This is splitting hairs though. It's a mistake in the code that caused it to do something unexpected that costs money. In the serverless world, that means invoking a function repeatedly, costing money. In the old-server world, maybe it means your script had a bug that downloaded an image repeatedly, causing you to rack up networking charges.

It is a mistake, yes. But this particular mistake would have behaved very differently on a normal server. Just because there exist mistakes you can make that would have the same consequences on regular server vs serverless doesn't mean you can just shrug your shoulders and say all mistakes are the same. The fundamental issue here is serverless is great at allowing you to automatically scale to meet demand, but it als…

Isn't there also a third danger with anything that scales your bill as your app scales - the possibility of some black hat ddos-ing you for the hell of it?
Post reply on HN