I'm curious what would've happened had the bill been for 10k or 1k? Would it still have been reduced, would OP have paid instead of posting on reddit/hn, would it have gotten as much attention?
Netlify just sent me a $104k bill for a simple static site
621–630 of 836 posts
Re: Netlify just sent me a $104k bill for a simple static site
#622Earlier quoted context omitted.
> I'd hate to be stuck with that $5K bill, I dunno, that does feel closer to the mark of something that maybe you should be on the hook for if you're responsible for 200 TB of bandwidth overage over 4 days? The responsibility part is the tricky part of the equation. If someone hits your site with a DDoS attack, are you responsible? There's literally nothing[0] you can do as a customer of a cloud provider here because…
All fair points but do they apply to the Netlify situation? As I understand it they generally won't hold you liable for resource usage generated by a DDoS, the guy on Reddit said this was a DDoS, the Netlify CEO said the traffic "didn't match attack patterns..." I think telling a free tier customer that they owe $104K was a pretty stupid PR move either way, but we don't really have enough info to say whether this was…
> As I understand it they generally won't hold you liable for resource usage generated by a DDoS
From personal experience as a customer of a cloud provider (not with Netlify btw), usually cloud providers who profit from bandwidth costs will write their TOS in such a way where almost nothing qualifies as a DDoS attack unless it's truly a distributed and targeted large scale attack specifically on your site.
A random person on the internet who spins up a few VPSs around the world and slams your site with looped curl requests won't count as a DDoS attack even though from your perspective that will result in a massive bill increase due to bandwidth costs.
In other words, I'm not surprised "didn't match attack patterns" was used. I'm guessing that will be the case most of the time.
Re: Netlify just sent me a $104k bill for a simple static site
#623Netlify CEO here. Our support team has reached out to the user from the thread to let them know they're not getting charged for this. It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact. Apologies that this didn't come through in the initial support reply.
So netlify is a major scammer organization now!? Uh oh time to look elsewhere
Re: Netlify just sent me a $104k bill for a simple static site
#624Earlier quoted context omitted.
There are only two questions everyone have: 1. Would Netlify forgive the bill if this didn't go viral? 2. How do you plan to address this issue so that it never happens again? Everyone here knew someone from Netlify would come and say OP wouldn't have to pay. That was a given. Now we want to know the important answers.
1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral 2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
You should probably consider a daily limit (up to some max n days) rather than a hard one time limit. If your engineers can set a 1 and done they can set an n and done and it would be a much better solution and more customer friendly. The guy using 5 gigs today as a poor college student will likely have a position in a small to mid-size company in a few years. I assume non-free (but low tier) customers would much prefer a reasonable limit set as well. Maybe a max of 2x (or so ) bandwidth so no huge surprises. Remember they're your customers and not your paying adversaries
Re: Netlify just sent me a $104k bill for a simple static site
#625Earlier quoted context omitted.
This is something I really like about Nearly Free Speech.net. Their model is that you deposit funds up front, and they will deduct from those funds as you use services. It helps that they actually are nearly free so that a single $20 deposit can last for months or years in many cases. It's bizarre to me that more services don't support billing this way, since there are tons of situations where I would much rather hav…
Yes it’s nice like that. Specially for side projects on AWS that could go wrong on your personal credit card. Also I heard they forgave bills sometimes.
Amazon will, but they also gauge their discount in how many prevention and security measures from their 5 Pillars you follow in your environment.
You can do stuff like "disallow any of these instances to be used in your env", so if you never use graphics cards, disallow the whole class.
You can also set limits like "no more than 20x m5.4xlarge".
But again, AWS is the worst about no actual hard limits, cause each system generates bills. Ive also seen the hell of "hidden system AWS Billing doesnt have is still submitting billing and we dont know what it is". Again, AWS enables basically infinite liability.
Ive also discussed with C levels that "every engineer and dev with AWS logins have an unlimited credit card to of which you're on the hook for". Lets just say that 'heartburn' doesnt even begin to describe the terror on their faces.
Re: Netlify just sent me a $104k bill for a simple static site
#626Earlier quoted context omitted.
> Cloudflare's ddos protection Yeah, we got hammered once with over 10TB/mo and noped out of Netlify as fast as we could: https://twitter.com/rethinkdns/status/1370342245841342466 Had to pay the bill in full. Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0. Makes it hard to move to any other platform. As a small tech shop, this is my Hotel California I'm happy to never leave.
That’s a free tier that doesn’t sound sustainable then, so that raises alarm bells to me.
I think a lot of people don't understand how cheap bandwidth is and is decreasing in cost practically every day. Amazon and Google have a lot of people fooled. Go ask someone operating in China and East Asia (and Japan) how much they're paying for local solutions.
Re: Netlify just sent me a $104k bill for a simple static site
#627Earlier quoted context omitted.
As far as I know, most cloud hosts don't actually support automatically moving live VMs, so I think it's fairly common for a host to be left running a single VM. At least in AWS, they never supported this, and in fact may require you to reboot an instance occasionally in order for it to be moved to a new hardware host (typically when they are upgrading their hardware).
But why are you talking about moving VMs?! Looks like you're adding tons of far-fetched speculations at every step of your reasoning. The way you easily deal with this issue is very simple and does not require moving VMs: you just allocate newly spawned VMs to existing hosts with available room! When you do so (and they obviously all do!) you end up with little unused hardware…
Say you have 3 hosts, each with a capacity of 10 VMs. At some point you have 28 running VMs - 10 on host1, 10 on host2, 8 on host3. Someone then closes down 2 of the VMs on host1, and 7 of the VMs on host3.
Now you have 19 VMs running, but need to keep all 3 hosts powered. If you don't have live VM moving, you are now forced to keep Host3 running only because 1 VM is running on it, even if that VM is idle. So, this one idle VM is responsible for all the energy consumption of host3, and will continue to be so until at least 3 more VMs get started (since you have room for 2 more VMs on host1).
If you did have live VM migration, or if the idle VM were powered down instead of running idle, you could close host3 completely, moving the VM to host 1, and only re-open host3 if needed for new VMs.
This is equivalent to the problem of memory fragmentation. Even though overall usage is low, if host usage is highly fragmented and you aren't allowed to move used memory around (compacting), you can end up consuming far more than actually needed.
Re: Netlify just sent me a $104k bill for a simple static site
#628I guess I never want my personal blog/site to be successful. Can't trust that I will wake up to a huge bill from EC2 or my CDN provider. Feels like health insurance in that even when you have insurance, you are never 100% confident you will not get some huge hospital bill. The pessimistic part of me says this is all deliberate to prevent the "little guy" from competing these days.
If you run on a single EC2 instance and you aren’t running an auto scaling cluster or anything of the sort, it would be pretty hard to get a huge bill. I much prefer that and the chance that it goes down then autoscaling or severless. Most serverless solutions have also gotten so config heavy or complex to make changes that most projects feel much better to me on an instance I can ssh into and poke around without hav…
Thanks, I use a "small" EC2 instance for my personal stuff (about $35 a month), with the Cloudflare free version, but I honestly don't know and what would happen if something went viral or a DOS. Would the bill be double? Triple? Would the site simply crash. What does Cloudflare do? I honestly have no idea.
Re: Netlify just sent me a $104k bill for a simple static site
#629Earlier quoted context omitted.
That’s a free tier that doesn’t sound sustainable then, so that raises alarm bells to me.
it's 100% not sustainable. Use it while it's good, but don't get vendor locked in, because sooner or later they will increase the prices
This is why we still use services on VM's and open source containers. We can move our services anywhere, including selfhosting. AWS and Google offer some amazing solutions, but lock in ain't worth it if you can manage your own stack via serverless/vm solutions.
Re: Netlify just sent me a $104k bill for a simple static site
#630Earlier quoted context omitted.
Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down. But that's on the user. The user shouldn't get upset in that scenario and has no right to. You're giving the control back to the user.
The infantilization of the user is common in tech now. For good reasons? Maybe. But it is common.
The site user/admin saying "If this spend goes over $100, shut shit down" is called being a fiscally responsible adult.
The fact that most cloud operators don't have actual hard cutoffs to maintain financial responsibility is intentional. Azure does, but only for specific account types. If it's PAYG, you can't do it. The end result is if you do something "weird", or someone DDoS's you, you're liable.
With a hard limit, a DDoS just takes your site offline.