Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 348 posts

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#181

Post by horsawlarway »

Mo3 wrote: post #80
sirwitti wrote: post #74

I'm very much on the boring technology side of things with respect to hosting. 40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily. If you also use containers you even get quite a bit of flexibility and agility. To be honest I don't really understand the sentiment that developers can get away with not knowing…

You're going to understand where the cool kids come from once your single dedicated server goes down or can't handle the load any more. As soon as you try to scale horizontally or become highly available and start to think about how to do it you end up falling into the same rabbit hole. > If you also use containers you even get quite a bit of flexibility and agility. Yeah... and then the only difference is between a…

> With all due respect and no offense intended, your perspective sounds a lot like "I've never attempted to scale so I can't understand the problems"

With all due respect, I don't think you've ever actually put together a local cluster.

A simple 4 machine k8s cluster sitting literally on dirt in my basement can scale out to the equivalent of thousands of dollars of AWS spend a month. I broke even on the initial purchase outlay for my workloads in less than a year.

The problems are almost never scaling the web servers. The problem is scaling the infrastructure that those servers need to be useful.

Generally - your DB is the first pain point, your network is the next.

If you can run it in a container, that service probably isn't the bottleneck for scaling, it's going to be whatever is providing the persistent disk for that service, and the network between the two.

Both of those things happen to also be fairly expensive to scale in the cloud as well.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#182

Post by ghaff »

closewith wrote: post #62
joatmon-snoo wrote: post #54

Earlier quoted context omitted.

If there were hard limits, that would also mean that the billing system is on the critical path for all systems, and not just an after-the-fact ETL.

Not necessarily. A cloud provider could retroactively cap charges at the hard limits, but only cut access to resources asynchronously. That's effective what happens when you complain now with AWS.

If AWS wanted to, they could absolutely implement a hard cap. It's not like letting some services run for a few hours until billing catches up costs them a lot of money.

What is true--not necessarily in order is:

- I suspect AWS in aggregate probably makes a fair bit of money on overages that a user eats but would have had a hard circuit in place if they could have, and

- Even reasonably designed hard circuit breakers (e.g. we cut off access to your stateful data unless you pay your bill but we won't delete it for 30 days) are still giving developers a potentially well-hidden foot-gun for a production environment that management might not actually want.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#183

Post by Mo3 »

horsawlarway wrote: post #181
Mo3 wrote: post #80

Earlier quoted context omitted.

You're going to understand where the cool kids come from once your single dedicated server goes down or can't handle the load any more. As soon as you try to scale horizontally or become highly available and start to think about how to do it you end up falling into the same rabbit hole. > If you also use containers you even get quite a bit of flexibility and agility. Yeah... and then the only difference is between a…

> With all due respect and no offense intended, your perspective sounds a lot like "I've never attempted to scale so I can't understand the problems" With all due respect, I don't think you've ever actually put together a local cluster. A simple 4 machine k8s cluster sitting literally on dirt in my basement can scale out to the equivalent of thousands of dollars of AWS spend a month. I broke even on the initial purch…

> With all due respect, I don't think you've ever actually put together a local cluster.

I have, and still do. It's neither redundant nor highly available. The power source isn't, the internet connection isn't and it's also located in my basement and not multiple regions.

> The problems are almost never scaling the web servers. The problem is scaling the infrastructure that those servers need to be useful.

There you go..

> Both of those things happen to also be fairly expensive to scale in the cloud as well.

No. [0]

[0]: https://i.ibb.co/TKmB9HX/image.png

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#184

Post by huksley »

mfrye0 wrote: post #178
huksley wrote: post #153

Earlier quoted context omitted.

Thank your for ECS suggestion. I am definitely considering it, but struggling to choose between ECS, Elastic Beanstalk or EC2. My past experience with ECS was a bit frustrating because I was forced to use CodeCommit to deploy and I didn't liked that. I would prefer to deploy directly from CI, for example from GitHub actions.

ECS runs on EC2. You basically register available servers and ECS automatically puts containers onto the instances where space is available. We have it setup with Github actions to automatically deploy to ECS as well.

I think I should definitely consider it. I was using ECS Fargate on another project and it was not using EC2.

Why you choose ECS with EC2 instances instead of Fargate?

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#185

Post by huksley »

mfrye0 wrote: post #139
mfrye0 wrote: post #133

I started off using Lambda as well and made the same sort of mistake. I can't remember exactly how much my bill was, but it was enough that it would have drained all my savings and effectively kill my startup. AWS was kind enough to write off most of it. We now use Lambda only for simple cron / background tasks, or consuming from Kinesis. We use ECS for everything else. ECS is nice because it's relatively simple comp…

On a side note, I don't know if you've already acquired any AWS credits. If not, Product Hunt Founders Club is a decent deal that will give you $5k in AWS credits. Between that and the no Stripe fees for 1 year, it paid for itself in no time. https://www.producthunt.com/founder-club

Thank you!

I recently applied for AWS Activate credits because our startup was a part of YCombinator Startup School recently.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#186

Post by stevewatson301 »

jlmorton wrote: post #171
bilekas wrote: post #166

Earlier quoted context omitted.

> your cost for doing a couple million, 2-second-long requests is about four cents. This seems wrong to me ? Can you explain a bit more ? Are these just API requests or ?

It's admittedly a simplification and a best case. For AWS Lambda, the price is in GB-seconds, and the amount of CPU available to your function is itself a function of the memory allocated. The price is $0.0000166667 for every GB-second on X86. So it looks like I've also misplaced a decimal. It's $0.20 per 1M requests with 1GB of memory. Lambdas can be sized from 128MB to 10GB, and pricing depends on the resources all…

Lambda function endpoints[1] are a replacement for some use cases.

[1] https://aws.amazon.com/blogs/aws/announcing-aws-lambda-funct...

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#187

Post by Fnoord »

cft wrote: post #137
jpalomaki wrote: post #116

Earlier quoted context omitted.

Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on. Good platform-as-service can solve many things for you and let you focus on the core thing you are pro…

I ran a website with 37 million users and 3.6Gbps peak bandwidth (JavaScript+thumbnails, no video) from my own two racks of Linux servers thay i have not systematically updated for years. The OSes were beyond the lts support winows. I manually compiled my own updates, but very rarely and only those that i deemed critical. Granted, the site stack was completely custom so the standard automated hacks didn't work. In 15…

With unattended-upgrades you might get away with such behavior. Also, spme webservers have a great track record. Then its the question if your other services are secure, and your JS/CSS.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#188

Post by Fnoord »

cft wrote: post #175
booi wrote: post #167

Earlier quoted context omitted.

I dont think this should be standard. It sounds like you’re saying you ran a service with 37 million people’s information on a software stack that was so old that not even the vendor is supporting it anymore and could be riddled with security issues that you wouldn’t even know much less be able to detect? It may work but certainly not going to get any security certifications this way..

I hardly stored any PII. Also, this was a high profile site that you know: if it was hacked, they would probably try to deny access and extort: it would make a lot of monetary sense. We would also have lost our merchant accounts very quickly (although CC numbers were not stored, only MD5s, but i suppose they could have been captured from the application's memory after TLS decryption but before MD5 hashing, although t…

Do you save logs? If yes, there's your PII.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#189

Post by stevewatson301 »

throwaway894345 wrote: post #160
sirwitti wrote: post #74

I'm very much on the boring technology side of things with respect to hosting. 40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily. If you also use containers you even get quite a bit of flexibility and agility. To be honest I don't really understand the sentiment that developers can get away with not knowing…

If you're putting your service behind CloudFront, you're probably aiming for DDOS protection and low latency or something that you can't easily get from a VPS. Of course, you can put your VPS behind CloudFront, but you would run into the same issue (albeit the fixed capacity of your server would effectively cap your bill). A single VPS is great if you don't need reliability or scale, but if you care about those thing…

Cloudfront is terrible for DDOS protection though, given that there is a per request pricing (and no, AWS WAF doesn't help as it just imposes additional pricing on top, the only thing it does is to prevent Cloudfront from processing those requests.)

However, if your statement is generally about CDNs such as Cloudflare or Bunny.net that don't have per request pricing, it makes sense.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#190

Post by judge2020 »

riknos314 wrote: post #118
ryandvm wrote: post #52

Earlier quoted context omitted.

I've heard they have a generous foot gun billing policy and thankfully I've never had to find out, but we shouldn't be that grateful, because ultimately the cloud providers do this in their own rather dishonorable self interest. It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss…

> It would be fairly simple for them to allow users to set up hard billing limits. former AWS SDE here I don't believe it would be "fairly simple" to build a completely new off switch into 150+ services, likely with multiple integration points in each service. In addition, the mere existence of an off switch introduces new failure points, where failure directly turns into downtime. The effort to implement this is far…

IMO Google Cloud has the solution for this - access to APIs is off by default and you must enable API access before anything will work. Their portal is pretty good at estimating costs in the first place, so resources created there aren't much of an issue, but having to use the portal to enable programmatic access is a great way to avoid mistakes.
Post reply on HN