Live data from Hacker News

Serverless Horrors

serverlesshorrors.com

461–470 of 503 posts

Re: Serverless Horrors

#461

I don't understand why it should be called "serverless" when using cloud infrastructure. Fundamentally you're still creating software following a client-server model, and expecting a server to run somewhere so that your users' clients work. To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an In…

A "Server" is typically a single machine that has a specific OS and runs layers of various software that allows your business logic to be accessed by other computers (by your users). For a "Server" you typically have to choose an OS to run, install all the support software (server monitoring, etc), update the software, and if the server fails you have to fix it or rebuild it. With "Serverless", your code is in a "fun…

I understand the underlying reasoning. I just don't like the terminology. Hence, "I don't understand... should be", rather than "... is". I think it's wrong that people end up using words like that. Like, almost on a moral level.

More generally, I don't like that a term ending with "-less" marks an increase in system complexity.

Re: Serverless Horrors

#462

Earlier quoted context omitted.

run entire aws infra locally while studying for aws certification?

Let’s rephrase the question then, why makes an application dependent on AWS?

In general that would be a good question, but you've asked it in a case where "use AWS" is the _only_ way to accomplish the goal... which is learning AWS.

Re: Serverless Horrors

#463

Earlier quoted context omitted.

I’ve accidentally hit myself with a bigger than expected AWS bill (just $500 but as a student I didn’t really want to spend that much). So I get being annoyed with the pricing model. But, I don’t think the idea of just stopping charging works. For example, I had some of their machine image thingies (AMI) on my account. They charged me less than a dollar a month, totally reasonable. The only reasonable interpretation…

> But, I don’t think the idea of just stopping charging works. You don't stop CHARGING. You stop providing the service that is accumulating charges in excess of what limit I set. And you give some short period of time to settle the bill, modify the service, etc. You can keep charging me, but provide a way to stop the unlimited accrual of charges beyond limits I want to set. > No, I think cloud billing is just inheren…

So you want to proactively determine if, at the current rate charges are accumulating, the budget will be exceeded?

That _also_ runs into problems!

Take, for example, a nightly job that spins up a few giant instances to do some batch processing and shuts them down. Running an hour a night, over the course of the month that's going to accumulate ~$300 in charges. Great, we can set a $400/mo budget and have some wiggle room and all is well!

But how can AWS know that you're going to shut the instances down? Looking only at the rate charges are accumulating, the first night those instances start up you are on track to run up a $7,000 bill!

So do we set a $400/mo budget and then just kill the account so it stops accumulating charges when we hit $400, or do we set a $7,000/mo budget to account for the potential rate of accumulation and risk exceeding our budget by 2,000%?

It would be nice if this were in fact just overcomplicating things, but after much thought and many arguments on the internet I really can't see an easy "general" solution to this. The solution is heavily dependent on your specific workload and usage patterns, and the tooling is there to manage that if you want: Create billing alerts, and run code to adjust your usage in response to them.

That all said: I would fully support some sort of "developer sandbox" account that allowed a "kill the account" billing limit. I'd really prefer it had some sort of obvious limitation to avoid people accidentally using it for production workloads or dev workloads turning into production ones. Something like a hard limit that shuts the account down in 30 days, or limiting inbound connectivity to only via a VPN or something. That's purely self interest though--I don't want to see the article on the top of HN every few weeks about how "Amazon killed my startup" because someone set a billing limit and then all their customers' data was deleted.

Re: Serverless Horrors

#464
post #309

Earlier quoted context omitted.

What about the space you're using? Do they delete it? Remove all your configurations? Prevent you from doing anything with your account until you up your limit or wait until your month resets? If you're worried about getting a big bill, and you don't care if it gets shut off when you're not using it, why don't you shut it down yourself? AWS made the tradeoff to keep the lights on for customers and if there is a huge…

How many times has AWS refunded you a five figure bill? I've heard stories from people who got refunded but were told that it would be the first and last time they would get a refund.

I think I'm up to two five figure bills and two six figure bills refunded for various companies/clients. On one account, we had about $70k refunded, then a year or two later $130k. The normal monthly spend was closer to $30k.

There were no warnings or "don't do it again". They, very reasonably IMO, asked us to essentially explain how and why this happened and how we'd stop it happening again. They then provided some additional guidance and resources around those areas. In the one case where the charges were due to compromised credentials, they asked us to rotate all of our access keys before they issued the refund.

Completely anecdotal and slightly dated information, but that's been my experience.

Re: Serverless Horrors

#465

When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony). Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had the…

> I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion They refunded you $100k with few questions asked, and you hate them for it? I’ve made a few expensive mistakes on AWS that were entirely my fault, and AWS has always refunded me for them. I imagine if Amazon did implement “shut every down when I exceed my budget” there’d be a bunch of horror stories like “I…

> horror stories like “I got DDOSed and AWS shutdown all my EC2s and destroyed the data I accidentally wrote to ephemeral storage.”

I mean, S3 also incurs ongoing charges, so if you're going to stop accruing charges you'd also be deleting your data that wasn't on ephemeral storage...

And potentially deleting all of your DNS zones (and recreating them will likely give you different nameservers so you'll need to wait for the registrar to update them once you're back)...

And...

Re: Serverless Horrors

#466
post #258

Earlier quoted context omitted.

Using that logic, isn’t it the fault of the user to set up an app without rate limiting?

It's misleading to promote a free tier that can then incur huge charges without being able to specify a charge cap.

If it can incur any charges at all then it isn't free.

Re: Serverless Horrors

#467
post #425

Earlier quoted context omitted.

Yes I’m sure large corporations and even startups are going to leave AWS because a few junior devs didn’t do their research.

You do know that large corporations and startups employ junior devs as well, right? All else being equal, would you rather choose the platform where a junior dev can accidentally incur a $1M bill (which would already bankrupt early startups), or the platform where that same junior dev get a "usage limits exceeded - click here to upgrade" email?

Well, first I wouldn’t give a junior dev with no experience admin rights to an AWS account and would I have tight guardrails around what they can do - like I’ve done now with over a dozen implementations for clients since I’ve been in consulting for five years and the four years before that as an architect for product companies.

I also wouldn’t give a junior dev access to production databases.

Also from working with AWS from both the inside (Professional Services) and the outside at a third party consulting companies, I know how aggressively AWS is about keeping startups and they would never risk losing the continuing revenue of a company like that.

Re: Serverless Horrors

#468
post #425

Earlier quoted context omitted.

Yes I’m sure large corporations and even startups are going to leave AWS because a few junior devs didn’t do their research.

You do know that large corporations and startups employ junior devs as well, right? All else being equal, would you rather choose the platform where a junior dev can accidentally incur a $1M bill (which would already bankrupt early startups), or the platform where that same junior dev get a "usage limits exceeded - click here to upgrade" email?

> All else being equal, would you rather choose the platform where a junior dev can accidentally incur a $1M bill

If a junior dev has the access to do that, then there is a big failure (probably more than one) by someone who isn't a junior dev after choosing AWS that was necessary to enable that.

Re: Serverless Horrors

#469

Earlier quoted context omitted.

How do you take your S3 service offline when they charge for storage or your EBS volumes? Your databases?

Block access to the service until the next billing period starts, or the user upgrades to a paid tier.

And it is still incurring charges for storage costs.

Re: Serverless Horrors

#470
post #309

Earlier quoted context omitted.

How many times has AWS refunded you a five figure bill? I've heard stories from people who got refunded but were told that it would be the first and last time they would get a refund.

I think I'm up to two five figure bills and two six figure bills refunded for various companies/clients. On one account, we had about $70k refunded, then a year or two later $130k. The normal monthly spend was closer to $30k. There were no warnings or "don't do it again". They, very reasonably IMO, asked us to essentially explain how and why this happened and how we'd stop it happening again. They then provided some…

Thanks for the anecdotes!
Post reply on HN