Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 348 posts

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#111

Post by turtlebits »

jtwebman wrote: post #26

For small projects why do you need the scale? I feel like once you need the scale serverless is the way more expensive then even managed Kubernetes. I still think serverless is hosting services way to make far more money with the illusion that it is easier when it really isn't. Logging is normally a huge pain. Local dev is usually a huge pain. Managing versions is a pain over just git branches especially over multipl…

Serverless isn't just about scale, it's about deploying code without having to touch any infrastructure. The lambda free tier is also very generous (1M free requests per month).

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#112

Post by Mo3 »

zamadatix wrote: post #106
Mo3 wrote: post #80

Earlier quoted context omitted.

You're going to understand where the cool kids come from once your single dedicated server goes down or can't handle the load any more. As soon as you try to scale horizontally or become highly available and start to think about how to do it you end up falling into the same rabbit hole. > If you also use containers you even get quite a bit of flexibility and agility. Yeah... and then the only difference is between a…

For 99% of projects you're never going to hit the point a single server (or group of servers if you truly need redundancy for some level of uptime) can't handle the load. For the other 1% that end up needing that scale I have a hard time accepting it's actually better to start building for massive scale day 1 instead of day 1000.

This is such a narrow perspective. It’s not always about load, most applications can benefit from some level of redundancy and using AWS etc doesn’t equal a massive scaling operation forced upon you. In fact, the other options would. I'll give you some common examples.

You have a script that needs to run, without fail, at a certain time every day.

You have an endpoint that is mission-critical and even a second of downtime would cause insane manual workload.

You have an endpoint that needs to have as little end-user latency as physically possible.

These are all real examples from my work in fintech. The solution to all of them is a few dollars in cloud functions in Lambda and Lambda@Edge. Now imagine having to provision, orchestrate and maintain dozens of dedicated hosts all by yourself, simply for these handful scripts. The expense of employee time alone would make it absolutely idiotic to go that route and if anything this would equal massive scale day - but even worse, because you didn't even plan for it beforehand.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#114

Post by dabinat »

I think every developer has an AWS billing horror story.

My horror story is that my site allows users to upload videos and share them to a limited number of colleagues. When a user requests a video, a CloudFront URL is created that lasts a few hours.

I had not thought much about hotlinking because the link only lasts a few hours - what would be the point? Well, those few hours make a big difference when it’s linked on a high traffic website.

Turns out someone paid for the cheapest plan ($7) and uploaded two multi-GB files. They hotlinked them on a Vietnamese porn site and ran up charges of almost $10k.

I was alerted by Cost Anomaly Detector but it had already run up most of those charges (and the totals CAD listed were much smaller and made it seem like less of a problem, thus delaying my reaction). AWS, to their credit, waived the charges.

I had WAF already setup but it wasn’t very helpful for this type of thing. I could only block sites that I already knew about. I ended up going with a Lambda@Edge solution that validates the source site before allowing access.

Lessons learned: 1. Customers may abuse things in ways you didn’t predict 2. Cost Anomaly Detector has a delay and only kicks in once charges have accrued. It can save you from an insane bill but won’t save you completely from large bills. 3. AWS can be reasonable about this but the ball is entirely in their court.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#115

Post by tluyben2 »

Mo3 wrote: post #112
zamadatix wrote: post #106

Earlier quoted context omitted.

For 99% of projects you're never going to hit the point a single server (or group of servers if you truly need redundancy for some level of uptime) can't handle the load. For the other 1% that end up needing that scale I have a hard time accepting it's actually better to start building for massive scale day 1 instead of day 1000.

This is such a narrow perspective. It’s not always about load, most applications can benefit from some level of redundancy and using AWS etc doesn’t equal a massive scaling operation forced upon you. In fact, the other options would. I'll give you some common examples. You have a script that needs to run, without fail, at a certain time every day. You have an endpoint that is mission-critical and even a second of dow…

I work in fintech (banking and payments) and most of our clients aren’t even allowed to host on aws; they have to go for a local provider because aws doesn’t have a hosting hub in their country. Not sure what part you work in but this has never been a problem in the past 20 years with just servers, switches, load balancers etc.

I prefer aws over metal for these kind of setups, but for many other cases I definitely do not; just a dedi or a vps with docker or k8s and/or something like openfaas is enough for almost all startups and beyond. Making it literally impossible to make mistakes like OP. And when needed maybe failover or load balancing.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#116

Post by jpalomaki »

sirwitti wrote: post #74

I'm very much on the boring technology side of things with respect to hosting. 40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily. If you also use containers you even get quite a bit of flexibility and agility. To be honest I don't really understand the sentiment that developers can get away with not knowing…

Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on.

Good platform-as-service can solve many things for you and let you focus on the core thing you are providing.

Obviously not everybody needs to worry so much about the stuff mentioned above. If you are providing SaaS solution, there’s a good chance some customer will start asking these questions as part of their procurement process.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#117

Post by jvanderbot »

shepardrtc wrote: post #85
sirwitti wrote: post #74

I'm very much on the boring technology side of things with respect to hosting. 40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily. If you also use containers you even get quite a bit of flexibility and agility. To be honest I don't really understand the sentiment that developers can get away with not knowing…

> To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff and at the same time have to spend relevant amounts of time, energy and money to get up to speed with cloud solutions, k8s and so on. There was a thread here the other day on how DevOps has failed. And this should hopefully show everyone why DevOps is needed. Cloud infrastructure is complex and ne…

Or does it? The whole point of GP is that you don't need the complexity that would require devops staff to manage if you are not serving > 10s of millions of reqs in huge bursts with long stretches of dead time between.

I don't know, I'm curious about the actual requirements vs the marketing buy-in and resume building that may happen when designing the system.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#118

Post by riknos314 »

ryandvm wrote: post #52
hnlmorg wrote: post #22

Earlier quoted context omitted.

That's not really fair because while AWS does have a lot of issues, their refund policy isn't one of them. It's usually really easy to present a case for refunding accidental charges.

I've heard they have a generous foot gun billing policy and thankfully I've never had to find out, but we shouldn't be that grateful, because ultimately the cloud providers do this in their own rather dishonorable self interest. It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss…

> It would be fairly simple for them to allow users to set up hard billing limits.

former AWS SDE here

I don't believe it would be "fairly simple" to build a completely new off switch into 150+ services, likely with multiple integration points in each service. In addition, the mere existence of an off switch introduces new failure points, where failure directly turns into downtime.

The effort to implement this is far from trivial, removes resources from implementing other features that the really large accounts are asking for, and adds complexity with direct availability risks. It's not at all surprising they don't implement this.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#119

Post by Mo3 »

tluyben2 wrote: post #115
Mo3 wrote: post #112

Earlier quoted context omitted.

This is such a narrow perspective. It’s not always about load, most applications can benefit from some level of redundancy and using AWS etc doesn’t equal a massive scaling operation forced upon you. In fact, the other options would. I'll give you some common examples. You have a script that needs to run, without fail, at a certain time every day. You have an endpoint that is mission-critical and even a second of dow…

I work in fintech (banking and payments) and most of our clients aren’t even allowed to host on aws; they have to go for a local provider because aws doesn’t have a hosting hub in their country. Not sure what part you work in but this has never been a problem in the past 20 years with just servers, switches, load balancers etc. I prefer aws over metal for these kind of setups, but for many other cases I definitely do…

I don’t know about that, the hedge fund I work for is US-based and AWS can be fully SEC and FINRA compliant.

We also have a few dedicated servers, but mostly only for infrequently accessed data and logging that doesn’t need to be highly available.

I really can’t understand why this argument keeps coming up. Different solutions for different usecases. Yet anytime Kubernetes or cloud functions are discussed people come in and go like “hurr durr my single Hetzner dedicated server can do all of that and doesn’t have these problems”

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#120

Post by kif »

sirwitti wrote: post #74

I'm very much on the boring technology side of things with respect to hosting. 40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily. If you also use containers you even get quite a bit of flexibility and agility. To be honest I don't really understand the sentiment that developers can get away with not knowing…

Reading some replies here, it’s no wonder some startups go bankrupt while investing so much in infrastructure — all while not having enough users to justify more than a single dedicated server.

Scalability should be the last thing in mind if your product sucks.

Post reply on HN