Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless: A lesson learned the hard way

sourcebox.be

Post reply on HN
Print view View original

71–80 of 139 posts

Re: Serverless: A lesson learned the hard way

#71

Post by lsc »

ryandrake wrote: post #46
bluGill wrote: post #42

Earlier quoted context omitted.

Yes, but that isn't always the right answer. If your system is starting to cost "a lot" is that because of a bug (this case), or is it because your idea just "went viral" and you are not getting tons of paying customers signing up? If it is the latter you do not want any rate limiting, you want everything to scale as fast as possible (I hope there are no bugs on your end). Rate limiting means that your new customers…

It's almost as if... they should offer multiple options so customers could choose based on their business/hobby needs: 1. Warn me at $X but don't throttle me for any reason--I'll pay if I go viral 2. Warn me at $X and start throttling until I get to $Y at which point stop service and stop charging 3. Warn me at $X and stop service/charging immediately

The market has done this, in part, through segmentation.

When you are on shared hosting, the expectation is that you get shut off when you go over.

When you are on "unlimited" shared hosting, the expectation is that you and everyone on the server gets throttled when you go over.

When you are on a VPS, the expectation is that you will be throttled when you go over, and you will be throttled much less than with other options when your neighbor goes over.

With cloud, then, the expectation is that if you go over, you are charged more proportionately, but things continue to work.

Of course, this is a simplification, but I think it accurate enough to be useful.

I do agree that it would be better to choose your api/provisioning and node reliability separately from overage behavior, but most of these behaviors and expectations were based on traditions that were shaped by technical constraints.

To credibly say "we will keep you online and just charge you" you need a lot of spare capacity.

Throttling one customer on a shared host without impacting other customers used to be very difficult. It is still way easier to throttle one VPS customer, and easier stil to throttle that one customer when they have their own kernel and reserved memory; it is not as big of a deal as it once was, considering everyone now uses ssd, but systems that share page cache are notoriously difficult to setup such that light users don't impact heavy users.

Re: Serverless: A lesson learned the hard way

#72

Post by [deleted] »

ryandrake wrote: post #46
bluGill wrote: post #42

Earlier quoted context omitted.

Yes, but that isn't always the right answer. If your system is starting to cost "a lot" is that because of a bug (this case), or is it because your idea just "went viral" and you are not getting tons of paying customers signing up? If it is the latter you do not want any rate limiting, you want everything to scale as fast as possible (I hope there are no bugs on your end). Rate limiting means that your new customers…

It's almost as if... they should offer multiple options so customers could choose based on their business/hobby needs: 1. Warn me at $X but don't throttle me for any reason--I'll pay if I go viral 2. Warn me at $X and start throttling until I get to $Y at which point stop service and stop charging 3. Warn me at $X and stop service/charging immediately

[deleted]

Re: Serverless: A lesson learned the hard way

#73

Post by [deleted] »

ryandrake wrote: post #46
bluGill wrote: post #42

Earlier quoted context omitted.

Yes, but that isn't always the right answer. If your system is starting to cost "a lot" is that because of a bug (this case), or is it because your idea just "went viral" and you are not getting tons of paying customers signing up? If it is the latter you do not want any rate limiting, you want everything to scale as fast as possible (I hope there are no bugs on your end). Rate limiting means that your new customers…

It's almost as if... they should offer multiple options so customers could choose based on their business/hobby needs: 1. Warn me at $X but don't throttle me for any reason--I'll pay if I go viral 2. Warn me at $X and start throttling until I get to $Y at which point stop service and stop charging 3. Warn me at $X and stop service/charging immediately

[deleted]

Re: Serverless: A lesson learned the hard way

#74

Post by vlucas »

cbhl wrote: post #37
tschellenbach wrote: post #21

Earlier quoted context omitted.

that depends for 5$ it breaks after a certain level of traffic. for many applications its always better to spend $$ instead of things shutting down. (hosting is usually insignificant compared to people, revenue etc)

If $180 is significant, then this person is probably a student or on a budget, not a business. So they'd probably prefer the downtime.

Oh come on. If this is a side project, it's likely impacting his personal monthly budget. A sudden, unplanned, $180 expense on a personal budget can EASILY be significant - even if you make six figures.

Re: Serverless: A lesson learned the hard way

#76

Post by cddotdotslash »

eridius wrote: post #28
cddotdotslash wrote: post #17

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

This is splitting hairs though. It's a mistake in the code that caused it to do something unexpected that costs money. In the serverless world, that means invoking a function repeatedly, costing money. In the old-server world, maybe it means your script had a bug that downloaded an image repeatedly, causing you to rack up networking charges.

Re: Serverless: A lesson learned the hard way

#77

Post by chrisco255 »

chrisco255 wrote: post #35
eridius wrote: post #28

Earlier quoted context omitted.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

The developer should be writing unit tests for their code so they can avoid small mistakes like this.

Down vote for advocating for unit tests? That's just good practice in general.

Re: Serverless: A lesson learned the hard way

#78

Post by eridius »

cddotdotslash wrote: post #76
eridius wrote: post #28

Earlier quoted context omitted.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

This is splitting hairs though. It's a mistake in the code that caused it to do something unexpected that costs money. In the serverless world, that means invoking a function repeatedly, costing money. In the old-server world, maybe it means your script had a bug that downloaded an image repeatedly, causing you to rack up networking charges.

It is a mistake, yes. But this particular mistake would have behaved very differently on a normal server. Just because there exist mistakes you can make that would have the same consequences on regular server vs serverless doesn't mean you can just shrug your shoulders and say all mistakes are the same.

The fundamental issue here is serverless is great at allowing you to automatically scale to meet demand, but it also is great at automatically scaling to meet unexpected resource usage caused by errors (or poor design). And so this means a mistake on your end can cost you a lot of money, because the system thought that it was real demand.

Re: Serverless: A lesson learned the hard way

#79

Post by skewart »

foobarchu wrote: post #53
jzawodn wrote: post #36

Earlier quoted context omitted.

LOL WAT? How is it "serverless" if you're provisioning a server?

He's saying that the problem is with "serverless architecture", because the problem could not possibly have happened without the use of a serverless architecture (e.g. provisioning a server). The problem is exclusive to serverless.

> The problem is exclusive to serverless.

That's simply not true. You can accidentally run up huge bills with EC2 instances too. One typo in your CloudFormation templates could spin up a ton of reserved p2.16xlarge's.

Of course, if you consider EC2, and other AWS services, to be "serverless" too - you're not physically managing your own racks after all - then, yeah, fair enough, it is a problem exclusive to these "serverless" IaaS/PaaS providers.

Re: Serverless: A lesson learned the hard way

#80

Post by V3loxy »

godot wrote: post #70
Dolores12 wrote: post #3

"a $180 actual cost. I was left with a light headed feeling, it's a lot of money for me" people still play with fire. limit your losses, go with digital ocean or something for 5$/mo flat no matter what.

His previous blog post actually said that he moved away from the exact $5/mo digital ocean plan you're talking about, to this. I think the author meant to do this less of a "play with fire" way but more of experimenting with new tech way. But yes, I agree that for personal sites, running with your own money, you probably want to stick with something safer like the $5/mo digital ocean box.

This is indeed the case, playing around with new tech. I've been a happy customer with DO for years but my own website was the ideal case to try out the whole serverless thing. As my website doesn't get much traffic so it'd cost me next to nothing. I do agree with most of the comments here though and the $5 DO box is the safest choice. I might have been a bit too excited with the new things and failed to think logically :)
Post reply on HN