Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netlify just sent me a $104k bill for a simple static site

old.reddit.com

Post reply on HN
Print view View original

771–780 of 836 posts

Re: Netlify just sent me a $104k bill for a simple static site

#771

Post by jart »

zettabomb wrote: post #328
wielebny wrote: post #271

Earlier quoted context omitted.

This strategy may work for a (D)DoS that is targeted to an application layer, but won't work if the attack is designed to exhaust your bandwidth. Once you're receiving more traffic than you network cards can handle, it does not matter if you'll drop the packets with iptables or not. I was the target of attacks that caused Hetzner to terminate my contract. I was leasing physical servers there, so I assume the attacks…

These days it seems that DDoS attacks are often not targeted at bandwidth either, but rather packets per second. It is (apparently) much easier to exhaust routing capacity with an inordinate number of tiny packets than with a still large number of large packets. Cloudflare has some fun ways to deal with this [0]. [0] https://blog.cloudflare.com/mitigating-a-754-million-pps-ddo...

What they did to me was flood the Linux Kernel with TCP connections. That's why it's so important to block IPs in the raw PREROUTING table. You need to nip it in the bud before Linux starts allocating any memory to the attacker.

Re: Netlify just sent me a $104k bill for a simple static site

#772

Post by jart »

bravetraveler wrote: post #575
jart wrote: post #171

Earlier quoted context omitted.

Use a token bucket on your web server to catch abusive IPs and then blackhole them using `iptables -t raw -I PREROUTING -s ip -j DROP`. I know. I run https://ipv4.games/ which invites hackers to unleash their botnets, and the service runs on a small VM with only a few cores. It's been attacked by botnets with 49,131,669 IP addresses. There's no Cloudflare frontend or anything like that, because back when I used Cloud…

Eventually you're probably going to want an ipset, at least. Otherwise processing your chain will continuously cost more, and more, and more.

I just declare firewall jubilee every now and then, where I flush the iptables and let people try again. It's also because people usually only control the IPs they use temporarily, so I don't want someone innocent later on to be blocked from using the service because someone abusive used their IPs beforehand. But even if I didn't do this, it doesn't cost much for Linux to iterate over an array of blocked int32's. It's really only allocated TCP connection resources that are problematic.

Re: Netlify just sent me a $104k bill for a simple static site

#773

Post by cookiemonsieur »

natnat wrote: post #756
mathgradthrow wrote: post #722

Earlier quoted context omitted.

How on earth could I, as a customer, be sure that netlify hadn't paid someone to DDOS me? If I were in charge of a business like that, I would have that thought constantly...

This is insane conspiratorial thinking? How would being the only host that happens to get DDOSed constantly be a good business proposition?

If you charge them for the extra bandwidth usage, it is. Not saying it's morally right, but definitely something a shady business would do. There's nothing "conspiratorial" about that. You'd be surprised how many conflicts of interest Big Gov and Big Business find themselves in.

Re: Netlify just sent me a $104k bill for a simple static site

#774

Post by keeganpoppen »

aurareturn wrote: post #331
bobfunk wrote: post #317

Netlify CEO here. Our support team has reached out to the user from the thread to let them know they're not getting charged for this. It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact. Apologies that this didn't come through in the initial support reply.

There are only two questions everyone have: 1. Would Netlify forgive the bill if this didn't go viral? 2. How do you plan to address this issue so that it never happens again? Everyone here knew someone from Netlify would come and say OP wouldn't have to pay. That was a given. Now we want to know the important answers.

if only i had $1 for every time for every time someone asked this exact question on HN. yes, we all get it: easy question is askable and not answerable. you want a gold star?

Re: Netlify just sent me a $104k bill for a simple static site

#775

Post by Cthulhu_ »

rpgwaiter wrote: post #738
mathgradthrow wrote: post #722

Earlier quoted context omitted.

How on earth could I, as a customer, be sure that netlify hadn't paid someone to DDOS me? If I were in charge of a business like that, I would have that thought constantly...

Why go through that effort when they could just lie about site usage and say you incurred a bunch of traffic? Or make fake site "hits" from localhost? It's really the trade-off for using any cloud host. You are implicitly trusting the host, their monitoring tools, their billing system, and their customer support when things go wrong

Both of those could be exposed via an audit or a whistleblower, either of which would destroy the company and its reputation overnight.

Re: Netlify just sent me a $104k bill for a simple static site

#776

Post by darksim905 »

aurareturn wrote: post #331
bobfunk wrote: post #317

Netlify CEO here. Our support team has reached out to the user from the thread to let them know they're not getting charged for this. It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact. Apologies that this didn't come through in the initial support reply.

There are only two questions everyone have: 1. Would Netlify forgive the bill if this didn't go viral? 2. How do you plan to address this issue so that it never happens again? Everyone here knew someone from Netlify would come and say OP wouldn't have to pay. That was a given. Now we want to know the important answers.

You don't see VPS providers like Vultr forgiving bills like this, nor do they make the news. Granted they are not the same scope as Netlify, but still.

Re: Netlify just sent me a $104k bill for a simple static site

#777

Post by brnt »

croemer wrote: post #748
brnt wrote: post #702

Earlier quoted context omitted.

That sort of adversarial weaponization of credit cards is really a US thing. In most of the world, payment is not relevant to the question of liability: you create a debt, whether or not you can debit a card. So, looks like DigitalOcean is another name to avoid. Also: what's common? At the scale these companies so desire, small percentages are still thousands of users. Supporting them with what's ultimately a fairly…

Partially agree, yes the liability is independent of the payment. But in practice the one who has the money has the benefit of the status quo. The other party without money has to actively take legal action which is costly enough that it's often not pursued.

That again depends on jurisdiction. In much of the EU, this is not expensive and a thriving industry exists to support businesses to get bills paid.

Re: Netlify just sent me a $104k bill for a simple static site

#779

Post by littlestymaar »

tsimionescu wrote: post #661
littlestymaar wrote: post #651

Earlier quoted context omitted.

Except you don't have 3 hosts but 3 thousands, and during the time you're stopping the 9 VMs somebody else is starting 5 or 15 new ones! Yes it is similar to memory fragmentation in some way, but your argument is like saying an integer stored on the heap costs a full memory page! You realize that it's nonsense. Sure in extreme edge cases it can, but that's not a good metric to know the memory footprint of an integer!…

Of course, it's not gonna be common. But it will occasionally happen, in such a large data center.

Then you need to account for their low share in idle VMs when measuring how much electricity it is responsible for. If it's only the case for 1% of the idle VMs, then you need to count only 1% of the electric power of a host per idle VM (+ the small fraction of a host CPU power that an idle VM consumes). In any case, it's going to be very small (~$20/year)[1] and the “it costs them nothing” is a good approximation of that, or at least a much better one that assuming that the cost they charge you reflects an expense on their side (which is the point that was argued by rafram at the very start of this discussion.

[1]: let say 10W, which at $.2 per kWh[2], ends up costing $17.5 for an entire year!

[2]: electricity prices from [here](https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...) $.2 per kWh is slightly above the rates in California and Rhode Island, which is the highest in the US for industrial use.

Re: Netlify just sent me a $104k bill for a simple static site

#780

Post by Hetzner_OL »

livrem wrote: post #528
jug wrote: post #425

Earlier quoted context omitted.

I agree and also delete my account. The only "fix" here is to act like Hetzner and null route upon DDoS, price cap the thing, or offer unlimited bandwidth on the free tier like e.g. Cloudflare Pages. Uncapped but paid is a recipe for disaster and you'll always be subject to the will of the support staff when something happens. If they can grasp to a straw leading to suspicions that it's not in fact a DDoS attack, you…

How does price caps work on Hetzner? I never managed to figure that out from reading their price lists. It looks to me like they charge for each TB, and the only thing I can see is that you can set an email alert to go off when close to some threshold?

Traffic limits change depending on our products: See https://docs.hetzner.com/robot/general/traffic/ 1) dedicated servers: unlimited 2) cloud servers: changes based on package 3) colocation: changes based on product 4) managed servers: unlimited 5) managed vServers: 20 TB 6) Storage Boxes: unlimited We only calculate outgoing traffic. We do not count incoming and internal traffic. --Katie
Post reply on HN