Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 348 posts

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#331

Post by horsawlarway »

icedchai wrote: post #298
horsawlarway wrote: post #181

Earlier quoted context omitted.

> With all due respect and no offense intended, your perspective sounds a lot like "I've never attempted to scale so I can't understand the problems" With all due respect, I don't think you've ever actually put together a local cluster. A simple 4 machine k8s cluster sitting literally on dirt in my basement can scale out to the equivalent of thousands of dollars of AWS spend a month. I broke even on the initial purch…

I have a k8s cluster in my basement (concrete floor, not dirt.) It's fine for dev work, but unsuitable for production. The real problem is redundant, reliable internet: cable internet upstream bandwidth sucks after 20 - 40 megabits/sec, and can't get fiber here yet.

This is fair.

I've used both comcast and google fiber while running my basement cluster. I think on google fiber I would have said production usage would be fine - I would pretty reliably get 1gbs down and ~800mbps up.

I wouldn't have redundant net, but the upload speeds are fine.

On Comcast... Comcast is really kind of a joke. Even on expensive business plans they cap upstream at 35mbps in my area (down is 1gbps) and the fastest way to get throttled to actually try using that upstream capacity.

Comcast does give me redundant failover to 5g by default now, which I guess is nice, but it's not worth much when I can't send a ton of data over the line anyways.

So point taken - if you're going to have to rent a room in a major metro for fiber access anyways, you probably want to be paying for cloud services (sadly - you'll pay a lot, since outbound traffic costs are high, and if 30mbps upstream isn't doing it for you, you're gonna blow through the free 100gb fast)

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#332

Post by mnkmnk »

huksley wrote: post #159
em1sar wrote: post #7

It's really difficult for AWS or any other serverless provider for that matter, to achieve a kind of "bulletproof and safe user experience" across different offerings that encompasses everything that has to do with billing/monitoring/alerting and then also cover all kinds of potential customer scenarios (like the function calling itself, as one example). For example, it's totally understandable that the alarms can be…

Sorry, I might have been not very clear, but AWS billing alert for $300 have been triggered only when it have reached $1,484 in charges. If that alert triggered earlier (and $300 would have been triggered in an hour), my all accumulated charges would be only $400 not $4500. So the hard learning here is that CloudFront charges takes time to appear on your bill, up to 24 hours.

24 hours delay in 2022 is egregious.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#333

Post by devmor »

salawat wrote: post #322
devmor wrote: post #92

Earlier quoted context omitted.

There is a point where that becomes exponentially prohibitively expensive. At a former startup where I worked as senior engineer, that was our original approach. Then one weekend we tripled our userbase and horizontally scaling required massive changes in database architecture, sharding solutions, etc. "Just buy more servers and load balance" is the short painful route. Carefully planning out and taking advantage of…

>Then one weekend we tripled our userbase and horizontally scaling required massive changes in database architecture, sharding solutions, etc. You took the short easy route out of the gate instead of building for what you were planning to handle. This is the opposite of what I espouse. If you were going about it my way, you build that in from the ground up. Harder and a bit bumpier to roll out from the get go, but ha…

If you want to spend capital for an unpredictable eventuality that may never pass over taking advantage of another company's investment to scale cost... sure, that's definitely the hard route.

I wouldn't call it "good" or "intelligent", but it's definitely hard.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#334

Post by rozenmd »

rozenmd wrote: post #200
huksley wrote: post #140

Earlier quoted context omitted.

Yes, moving to VM is definitely doable, but now, being a 1 person dev team, it is challenging in maintenance. My fear is, being on a vacation, and suddenly this VM dies. It might take too much time to bring it back online, and I might be out of good network coverage.

Self healing VMs - look into fly.io, they just restart when out of memory etc took about a day to rewrite and a week of 30 mins per day to figure out how to optimise my code for self healing. knowing my bill is capped by number of VMs * memory, saves a lot of stress

In case you're curious, I wrote about my experience here: https://onlineornot.com/on-moving-million-uptime-checks-onto...

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#335

Post by TameAntelope »

perrygeo wrote: post #330
TameAntelope wrote: post #206

Earlier quoted context omitted.

How the fuck do you blow through $150k at an early stage startup???

Storage, compute and networking mostly. I'm confused by your question - are you suggesting that early stage startups couldn't possible generate a workload to warrant that cost? Maybe if your idea of a startup business is limited to running a basic CRUD app. But other data-intensive projects have engineering needs that can easily reach that scale, especially in the early days when you're trying to bootstrap your data…

Those cases are rare as hell, $150k should be enough for nearly anyone. You are making bad engineering decisions if you go through that in under a year.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#336

Post by cft »

Fnoord wrote: post #286
cft wrote: post #196

Earlier quoted context omitted.

Thankfully, the servers were not in Europe! I said "hardly". It's genuinely amusing how people downvote facts, simply because they don't fit the zeitgeist.

Nobody mentioned Europe, except you. If you are referring to GDPR this still applies if you host outside of Europe. We disagree webserver logs contain 'hardly any PII'. You may find it insignificant PII, but it does not constitute 'hardly'. IMO its a circumstantial goldmine.

In the US at least one court found that IPs were not PII

https://www.huntonprivacyblog.com/2009/07/10/washington-cour...

I am curious who "we" are- a royal plural?

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#337

Post by Fnoord »

cft wrote: post #336
Fnoord wrote: post #286

Earlier quoted context omitted.

Nobody mentioned Europe, except you. If you are referring to GDPR this still applies if you host outside of Europe. We disagree webserver logs contain 'hardly any PII'. You may find it insignificant PII, but it does not constitute 'hardly'. IMO its a circumstantial goldmine.

In the US at least one court found that IPs were not PII https://www.huntonprivacyblog.com/2009/07/10/washington-cour... I am curious who "we" are- a royal plural?

And in EU, it is.

We referred to you and me.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#338

Post by arlk »

impostervt wrote: post #4

I once committed my private AWS keys to a public github repo. A bot scooped it up nearly instantly and spun up many, many ec2 instances that were (probably) mining bitcoins. I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night. In the morning, when I learned what had happened, my bill was over $3k. I fixed the issue and emailed AWS asking for…

Consider yourself lucky. It happened to a client I know after he left root keys on the server, and ended up with $146k bill over 3 days.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#339

Post by rob_c »

Komodai wrote: post #289
rob_c wrote: post #78

Self-host and stop using the cloud, especially when you clearly don't know what you're doing. > Now I am waiting on a response from AWS Support on these charges; maybe they can help me waive part of that. Honestly why should they? They're very clear about their billing policy and pricing, the only reason they might is good PR/karma from yc posting...

> the only reason they might is good PR/karma from yc posting not really, they've been known to do it before, I personally got a much smaller charge (~$12) waived by just asking nicely.

That's directly akin to getting the good stuff until you're hooked...

Throw away the account, rent a bare metal server or even a self hosted vm if that's too expensive and you have far more freedom to learn things and do it properly.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#340

Post by bluecrab »

TameAntelope wrote: post #335
perrygeo wrote: post #330

Earlier quoted context omitted.

Storage, compute and networking mostly. I'm confused by your question - are you suggesting that early stage startups couldn't possible generate a workload to warrant that cost? Maybe if your idea of a startup business is limited to running a basic CRUD app. But other data-intensive projects have engineering needs that can easily reach that scale, especially in the early days when you're trying to bootstrap your data…

Those cases are rare as hell, $150k should be enough for nearly anyone. You are making bad engineering decisions if you go through that in under a year.

Obviously. Lmao.
Post reply on HN