Live data from Hacker News

Serverless Horrors

serverlesshorrors.com

331–340 of 503 posts

Re: Serverless Horrors

#331

Earlier quoted context omitted.

You haven’t been able to use your Amazon retail account to open an AWS account for years. You don’t “beg”. You just send them an email and they say “yes”.

They are economic realists about this. They say "yes" if they can't realistically get $100k from you for your error.

I have never in 9 years working with AWS - four at product companies as the architect, 3.5 at AWS itself working in the Professional Services department and the last two working at 3rd party companies - ever heard or read about anyone either on a personal project or a large organization not be able to get a refund or in the case of a large org, sometimes a credit from AWS when they made a mistake that was costly to them.

I haven’t even seen reports on subreddits

Re: Serverless Horrors

#332

The real serverless horror isn't the occasional mistake that leads to a single huge bill, it's the monthly creep. It's so easy to spin up a resource and leave it running. It's just a few bucks, right? I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice an…

Sounds like your organization isn’t learning from these periods of high bill. What lead to the bill creeping up, and what mechanisms could be put in place to prevent them in the first place?

> what mechanisms could be put in place to prevent them in the first place?

Those mechanisms would lead to a large reduction in their "engineering" staff and the loss of potential future bragging rights in how modern and "cloud-native" their infrastructure is, so nobody wants to implement them.

Re: Serverless Horrors

#333

Earlier quoted context omitted.

> But, I don’t think the idea of just stopping charging works. You don't stop CHARGING. You stop providing the service that is accumulating charges in excess of what limit I set. And you give some short period of time to settle the bill, modify the service, etc. You can keep charging me, but provide a way to stop the unlimited accrual of charges beyond limits I want to set. > No, I think cloud billing is just inheren…

So, are you looking for some “rate of charges” cap? Like, allow the charges to accumulate indefinitely, but keep track of how much $/sec is being accumulated, and don’t start up new services if it would cause the rate of charges to pass that threshold? Might work. I do think that part of the appeal of these types of services is that you might briefly want to have a very high $/sec. But the idea makes sense, at least.

A theme of many of the horror stories is something like "I set up something personal, costing a few dollars a month, and I was DDOSed or (in earlier terms) slashdotted out of the blue, and I now have a bill for $17k accumulated over 4 hours".

As someone else pointed out, some(?) services prevent unlimited autoscaling, but even without unlimited, you may still hit a much larger limit.

Being able to say 'if my bill goes above $400, shut off all compute resources' or something like that. Account is still on, and you have X days (3? 1? 14?) to re-enable services, pay the bill, or proceed as you wish.

Yes, you might still want some period of high $/sec, but nearly every horror story in this vein ends with an issue with the final bill. Whether I burn $300 in 5 minutes or 26 days, I want some assurance that the services that are contributing most to that - likely/often EC2 or lambda in the AWS world - will be paused to stop the bleeding.

If you could pipe "billing notification" SNS message to something that could simply shut off public network access to certain resources, perhaps that would suffice. I imagine there's enough internal plumbing there to facilitate that, but even then, that's just AWS - how other cloud providers might handle that would be different. Having it be a core feature would be useful.

I was on a team that had our github CI pipeline routinely shutdown multiple times over a few weeks because some rogue processes were eating up a lot of minutes. We may have typically used $50/$100 per month - suddenly it was $100 in a day. Then... $200. Github just stopped the ability to run, because the credits used were over the limits. They probably could run their business where they would have just moved to charging us hundreds per day, perhaps with an email to an admin, and then set the invoice at $4500 for the month. But they shut down functionality a bit after the credits were exhausted.

Re: Serverless Horrors

#334

I don't understand why it should be called "serverless" when using cloud infrastructure. Fundamentally you're still creating software following a client-server model, and expecting a server to run somewhere so that your users' clients work. To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an In…

A "Server" is typically a single machine that has a specific OS and runs layers of various software that allows your business logic to be accessed by other computers (by your users). For a "Server" you typically have to choose an OS to run, install all the support software (server monitoring, etc), update the software, and if the server fails you have to fix it or rebuild it. With "Serverless", your code is in a "fun…

> Essentially you do not have to deal with any of the hassle that comes with setting up and maintaining your own "server", all you have to do is write the code that is your business logic.

Also known as "shared hosting". It's been done since the 90's (your folder full of PHP files is an NFS mount on multiple Apache servers), just that the techbros managed to rebrand it and make it trendy.

Re: Serverless Horrors

#335

I tried AWS serverless, figured out that it is impossible to test anything locally while you are forced to use AWS IAM role for serverless run which has access to everything. That's just a problem waiting to happen while you are always running tests on production...

1. Put your stuff in a stack. Deploy it to your isolated developer account. Basically free staging environment. 2. Use the officially supported docker runtime for local testing. 3. Treat it like any other code and make unit tests 4. Use one of the tools like localstack to emulate your staging env on your machine. There are so many options that I don’t know how you could walk away with your opinion.

Or you could just write conventional software. But I get it, you don't get resume points nor invites to cloud-provider conferences for that.

> Basically free staging environment. [emphasis mine]

Not really. Sure, the cost would usually be peanuts... until you have an infinite loop that recursively calls more lambdas. Then you have a huge bill (but hey that pays for your invites to their conferences, so maybe it's a blessing in disguise?). And yes, you will pretty much always get it refunded, but it's still a hassle and something that is absolutely not necessary.

Snark aside, having an opaque dev environment always constrained by bandwidth and latency that can’t be trivially backed up/duplicated is a terrible idea and why I always recommend against “serverless”, even besides the cost concerns.

Serverless is OK for small, fully self contained pieces or code that are fire and forget. But for anything complex that’s likely to require maintenance, no thanks.

Re: Serverless Horrors

#336
An alternative title might be "Failure to read the documentation horrors."

If you didn't sit down with the documentation, the pricing guide, and a calculator before you decided to build something then you share a significant portion of the fault.

Re: Serverless Horrors

#338

Earlier quoted context omitted.

This story is giving "I leave OWASP top 10 vulns in my code because hacker mindset". It's not that hard to configure access controls, they're probably cutting corners on other areas as well. I wouldn't trust anything this person is responsible for.

It's about rate limiting, not access controls. Without implementing limits your spend can go above what your budget is. Without cloud you hit natural rate limits of the hardware you are using to host.

> It's about rate limiting, not access controls.

You just shouldn't be using S3 to serve files directly. You can run most public and many private uses through CloudFront. Which gives you additional protections and reduces things like per object fetch costs.

> you hit natural rate limits

Seen by your customers or the public as a "denial of service." Which may actually be fine for the people who truly do want to limit their spending to less than $100/month.

Re: Serverless Horrors

#339

I don't understand why it should be called "serverless" when using cloud infrastructure. Fundamentally you're still creating software following a client-server model, and expecting a server to run somewhere so that your users' clients work. To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an In…

Serverless is easier to say than "load controlled ephemeral server management." Which is the real point. As my load increases the number of allocated resources, like servers, increases, and as it decreases so do the allocations and costs.

This is great if you are willing to completely change your client-server code to work efficiently in this environment. It is a strain over a standard design and you should only be using it when you truly need what "serverless" provides.

Re: Serverless Horrors

#340

I tried AWS serverless, figured out that it is impossible to test anything locally while you are forced to use AWS IAM role for serverless run which has access to everything. That's just a problem waiting to happen while you are always running tests on production...

I use my AWS security key to run local tests. It works perfectly fine. You just need a ~/.aws/credentials file appropriately configured.

I have a makefile system which controls lambda deployments. One step of the deployment is to gather the security requirements and to build a custom IAM role for each individual lambda. Then I can just write my security requirements in a JSON file and they're automatically set and managed for me.

The real joy of AWS is that everything works through the same API system. So it's easy to programmatically create things like IAM roles like this.

Post reply on HN