Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 348 posts

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#291

Post by manishsharan »

drdaeman wrote: post #270
icedchai wrote: post #266

Earlier quoted context omitted.

You'd be surprised. I've worked at many startups and mid-sized companies. You don't actually "need" to do most of those things. I've did a consulting job for a company with 100's of millions in revenue that didn't patch its servers for years. One had a 1400 day uptime!

> didn't patch its servers for years. One had a 1400 day uptime Well, this isn't exactly the best practice to follow - but, yes, this is very common, and it works (until someone finds that there's a vulnerability to exploit). So, from a purely practical perspective - software updates are certainly not a hard requirement, one can do quite well without them (just be aware about the risks). Anyway, there are many fairly…

Hardening servers isn't as hard as it used to be. Ansible does a pretty good job and there are several examples of using it to harden your server. If you are using a VPS .. you could use packer and start with a fleet of servers based on your custom image.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#292

Post by pydry »

riknos314 wrote: post #118
ryandvm wrote: post #52

Earlier quoted context omitted.

I've heard they have a generous foot gun billing policy and thankfully I've never had to find out, but we shouldn't be that grateful, because ultimately the cloud providers do this in their own rather dishonorable self interest. It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss…

> It would be fairly simple for them to allow users to set up hard billing limits. former AWS SDE here I don't believe it would be "fairly simple" to build a completely new off switch into 150+ services, likely with multiple integration points in each service. In addition, the mere existence of an off switch introduces new failure points, where failure directly turns into downtime. The effort to implement this is far…

Strictly speaking even the simplest features spanning all of the services wouldnt be simple.

It's got 0 to do with why the feature doesnt exist though.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#293

Post by fartcannon »

moreira wrote: post #284
fartcannon wrote: post #257

Earlier quoted context omitted.

Starting a db on a web-server and running apt update/upgrade are fundamental parts of working on the web. If you or any of your developers don't know how to do that, you/they should seriously seriously spend the 40 minutes it takes to learn.

I feel like that’s where you separate the pre-cloud and post-cloud devs. It’s not hard to imagine a dev being brought up 5 years ago into a React world, with JS in the backend and something like Vercel or Netlify to deploy their app. Someone like that might be entirely uncomfortable when dumped into a plain old Linux server, and would definitely take longer than 40 minutes to even get their head wrapped around what’s…

Symapthy, is more like it. Theyve been convinced by the cloud rental services that it's different now and too hard, when it's really not because all that SaaS is still running on a webserver. There's just a new gate keeper with a huge marketing budget that charges by the minute now.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#295

Post by remram »

wooque wrote: post #193
jpalomaki wrote: post #148

Earlier quoted context omitted.

Take for example OS and app upgrades. Quite often you have few requirements, for example 1)all updates should be first tested in test env 2)updates need to be installed in timely manner 3)critical updates need to be installed quickly (30 days is too slow) When you start thinking these, they are not so easy. (1) means you can’t just run “apt upgrade” on every server - you need to manage the updates to make sure they g…

Small teams should just configure apt unattended upgrades and call it a day. I do this with my personal server and haven't had any issues for years.

You should probably reboot it then.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#296

Post by icedchai »

throwaway894345 wrote: post #271
icedchai wrote: post #268

Earlier quoted context omitted.

A 10ms invocation sounds like a myth. I've certainly never seen one that low for a lambda that actually does anything. All the serverless stuff I've worked with (mostly in Python) operates slower than a CGI script. 100's of milliseconds minimum. When your serverless endpoint needs to integrate with other AWS services, the configuration can get complicated fast. VPC endpoints, IAM roles, security groups... it goes on…

If your lambda runs very infrequently, you'll see more cold starts which will take hundreds of ms, but warm starts are pretty easily in the single-digit milliseconds (at least for a Go lambda). If you're doing a bunch of compute or sync I/O in a loop then your functions will take longer. But less frequent invocations are actually an even better case for Lambda versus a VPS because it suggests less wasted time (yeah,…

Yes, cold starts are definitely a problem. One of the advantages of VPSes is its all hot, ready to go, obviously. You pay for that performance, for that lower latency, and I'm fine with that.

My big problem with lambda / serverless is the developer experience is pretty awful. The time between making a change, deploying, and seeing the result of that change is slow. You can work around this (with tools like localstack), but it's often not close enough to the real environment. You'll still waste tons of time debugging permissions issues when you do a real deploy.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#297

Post by fswd »

sleepdreamy wrote: post #31

This is the exact reason I fear AWS. How should I go about learning it when mistakes like this can basically ruin me for the next few months? Not sure if there are safe resources or Free Sandboxes somewhere

Last weekend I went on a code marathon doing a bunch of CDK AWS stuff.

Soon as I saw this headline, I was completely shocked and dropped everything to check my usage. Luckily I had 'cdk destroy' the big projects, but I had dozens and dozens of lambda and S3 buckets it failed to clean up. I spent probably 1 hour clicking through the web interface deleting them in fear of what happen to OP

I think I'll just stick to LocalStack and VPS.... actually my datacenter friend said I could bring up my deep learning station to this DC might just abandon AWS idea.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#298

Post by icedchai »

horsawlarway wrote: post #181
Mo3 wrote: post #80

Earlier quoted context omitted.

You're going to understand where the cool kids come from once your single dedicated server goes down or can't handle the load any more. As soon as you try to scale horizontally or become highly available and start to think about how to do it you end up falling into the same rabbit hole. > If you also use containers you even get quite a bit of flexibility and agility. Yeah... and then the only difference is between a…

> With all due respect and no offense intended, your perspective sounds a lot like "I've never attempted to scale so I can't understand the problems" With all due respect, I don't think you've ever actually put together a local cluster. A simple 4 machine k8s cluster sitting literally on dirt in my basement can scale out to the equivalent of thousands of dollars of AWS spend a month. I broke even on the initial purch…

I have a k8s cluster in my basement (concrete floor, not dirt.) It's fine for dev work, but unsuitable for production. The real problem is redundant, reliable internet: cable internet upstream bandwidth sucks after 20 - 40 megabits/sec, and can't get fiber here yet.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#299

Post by krallja »

cft wrote: post #175
booi wrote: post #167

Earlier quoted context omitted.

I dont think this should be standard. It sounds like you’re saying you ran a service with 37 million people’s information on a software stack that was so old that not even the vendor is supporting it anymore and could be riddled with security issues that you wouldn’t even know much less be able to detect? It may work but certainly not going to get any security certifications this way..

I hardly stored any PII. Also, this was a high profile site that you know: if it was hacked, they would probably try to deny access and extort: it would make a lot of monetary sense. We would also have lost our merchant accounts very quickly (although CC numbers were not stored, only MD5s, but i suppose they could have been captured from the application's memory after TLS decryption but before MD5 hashing, although t…

MD5(CC#) is trivially reversed, so I hope that’s not what you were doing.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#300

Post by johnklos »

Think about it this way:

You're a small, three person group. Are you more worried about having a modest colocated server or a small VPS run out of resources because you suddenly became really, really popular, or are you more worried about who'll go without salary for a few weeks because you misconfigured something and got a bill for multiple thousands of dollars?

Amazon takes full advantage of the fact that you can get in to deep billing waters very quickly. They can't collect your billing info quickly enough? Bull - and I mean this emphatically - shit. If a company that runs some of the largest datacenters in the world can't communicate metadata in real time, they're either hopelessly inept, or they're failing on purpose to make more $.

Post reply on HN