Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless: A lesson learned the hard way

sourcebox.be

Post reply on HN
Print view View original

111–120 of 139 posts

Re: Serverless: A lesson learned the hard way

#111

Post by voganmother42 »

thorum wrote: post #39

I wrote my (small) AWS app so it can run both on AWS and my local machine. Then you can write tests against the higher-level logic like "save this file to S3" and run those tests locally as well. My main challenge with serverless is using Lambda with API Gateway. Lambda has no database connection pooling, so I end up with a ridiculous number of connections to RDS - one for each simultaneous user. I haven't found a so…

One solution, use external pooler like pgbouncer or mysql-proxy running on a small instance(s)

Re: Serverless: A lesson learned the hard way

#112

Post by dvfjsdhgfv »

sudhirj wrote: post #82
dvfjsdhgfv wrote: post #57

Earlier quoted context omitted.

I know you're being sarcastic but I feel it's partly true. They announce so many different services, each month something new appears, but this very basic feature - bill capping - asked by users from the very beginning, has never been implemented. It's hard to believe they lack the skill or that it would be much more complicated than the current alert system.

According to the folks I've spoken to, they don't do bill capping because they have no way to safely shut down your workloads in any way - they'd prefer to let you know and have you do it. And having that choice is way better than a capping operation destroying your production database or causing downtime for your users.

There are so many ways of solving this, and some of them have been suggested to Amazon. They do have some very competent engineers able to implement it in a way that would make everybody happy.

Re: Serverless: A lesson learned the hard way

#113

Post by amcleod »

Not sure if someone has mentioned this already, but you should contact AWS support and ask if they will forgive the bill given it was an error which led to the high costs. I’ve had bills forgiven this way in the past (e.g. forgot to disable an instance that wasn’t really doing anything).

Re: Serverless: A lesson learned the hard way

#114

Post by eridius »

antaviana wrote: post #106
eridius wrote: post #28

Earlier quoted context omitted.

It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?

There are chances that degradation or unavailability are not free as in beer. If the degraded or offline system is used by people, and these people cannot work, the cost can be a lot higher. For example, 10 people not able to work could cost something in the range of $250-$750 per hour. Moreover, if customers are lost due to this degradation of service and CAC is high, then clearly the cheapest thing is a high bill b…

Oh sure. That's why I posed it as a question. Service degrading or going offline could be disastrous and cause losses of thousands of dollars or more, depending on what the service is. But there's also plenty of services where it's cheaper to have the service go down than it is to get an outsized AWS bill. This is just something you need to be aware of when deciding if serverless is the way to go.

Re: Serverless: A lesson learned the hard way

#115

Post by traduz »

cutler wrote: post #100
eridius wrote: post #78

Earlier quoted context omitted.

It is a mistake, yes. But this particular mistake would have behaved very differently on a normal server. Just because there exist mistakes you can make that would have the same consequences on regular server vs serverless doesn't mean you can just shrug your shoulders and say all mistakes are the same. The fundamental issue here is serverless is great at allowing you to automatically scale to meet demand, but it als…

Isn't there also a third danger with anything that scales your bill as your app scales - the possibility of some black hat ddos-ing you for the hell of it?

Yes, but I guess in that case you would put your lambda function behind an API gateway, and limit the user requests. If it's a static content you would serve it from a CDN. Not a specialist on this, but that's what I would do.

Re: Serverless: A lesson learned the hard way

#116

Post by tomc1985 »

lostcolony wrote: post #105
dvfjsdhgfv wrote: post #59

Earlier quoted context omitted.

You don't really need to administer it, there are plenty of hosting options, not just bare metal. And you can always add Cloudlflare anyway.

You're missing my point. "Serverless" just means "Applications built on cloud services rather than server(s) I have to administer". The OP said I should run my own infrastructure. I -could- host my blog by running a web server atop a server I administer, sure. I'd have to take on all the infrastructural tasks of doing that, securing it, ensuring any availability/scalability concerns I may have are taken care of, etc,…

It's as simple as shared host FTP drop (for static content), or a cheap VPS on DO, Linode, or what-have-you. Takes all of an hour to configure (with some practice), and if you are running a flavor of linux with APT and ufw, that includes setting up a firewall and unattended-upgrades.

And Cloudflare is still an option, since everyone needs their precious caching.

Re: Serverless: A lesson learned the hard way

#118

Post by orthecreedence »

voganmother42 wrote: post #111
thorum wrote: post #39

I wrote my (small) AWS app so it can run both on AWS and my local machine. Then you can write tests against the higher-level logic like "save this file to S3" and run those tests locally as well. My main challenge with serverless is using Lambda with API Gateway. Lambda has no database connection pooling, so I end up with a ridiculous number of connections to RDS - one for each simultaneous user. I haven't found a so…

One solution, use external pooler like pgbouncer or mysql-proxy running on a small instance(s)

I'm actually kind of blown away RDS doesn't have pgbouncer installed on the database. That's how we operate...each db server has pgbouncer living right on it. We connected to bouncer, not directly to the DB.

Re: Serverless: A lesson learned the hard way

#120

Post by vacri »

SimonPStevens wrote: post #101
sudhirj wrote: post #82

Earlier quoted context omitted.

According to the folks I've spoken to, they don't do bill capping because they have no way to safely shut down your workloads in any way - they'd prefer to let you know and have you do it. And having that choice is way better than a capping operation destroying your production database or causing downtime for your users.

I'm sorry, I just don't buy that. It doesn't have to be a hard cap, it could be a soft one. i.e. at £x your servers start shutting down, you'll get billed for a few extra minutes over your cap, before things have finished shutting down. Servers are totally capable of being shutdown without destroying databases. Besides, we aren't really talking about production databases at large companies. The people who want caps a…

> The people who want caps are devs learning and experimenting.

Then AWS is not focused on their use-case. You can make the argument that they're throwing away potential business here, but AWS is already the gorilla in the room, and people clamour for their products already. A couple of years ago they were rated as being bigger than the next 17 VPS providers combined.

> I'm sorry, I just don't buy that. It doesn't have to be a hard cap, it could be a soft one. i.e. at £x your servers start shutting down, you'll get billed for a few extra minutes over your cap, before things have finished shutting down. Servers are totally capable of being shutdown without destroying databases.

The idea of a payment cap sounds easy, but with something as complex as AWS, it's incredibly difficult, and everyone would demand different behaviour at the cap. So, you hit your cap. Turning off EC2 servers is easy. What about the data you've stored on s3? That costs. Should it be purged? What about your disk drives on EBS, should they be purged? How about items you have queued in SQS, should they be purged? Are you using RDS databases? They can't be stopped, only destroyed (you can do a final snapshot, but that's going to go to block storage, which costs. not much, but it costs).

"Just stop anything that costs" sounds easy, but it's not, not when you have a service as complex as AWS. AWS's current model of "forgive the bill for obvious mistakes" is way more workable.

Post reply on HN