Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netlify just sent me a $104k bill for a simple static site

old.reddit.com

Post reply on HN
Print view View original

721–730 of 836 posts

Re: Netlify just sent me a $104k bill for a simple static site

#721

Post by 7moritz7 »

BigJono wrote: post #516
dang wrote: post #511

Earlier quoted context omitted.

This is in the FAQ: https://news.ycombinator.com/newsfaq.html . See "How are stories ranked?" and "Why is A ranked below B even though A has more points and is newer?" About this specific case: it set off the flamewar detector (a.k.a. the overheated discussion detector) and also got moderation downweights. We sometimes turn off that penalty, but I don't think we'd do so in a case like this, because HN gets so many po…

I don't really buy that to be honest. I read this whole thread before the CEO posted and after, and neither time thought any of the comments were out of line or even that the general mood was any more heated than any other random HN thread. People are politely asking pertinent questions. And I think once the CEO makes a statement which contradicts the company's support response, that becomes very interesting. Particu…

I'm not a fan of opaque ranking algorithms either. Just use this, the only thing it has is a 500 point threshhold and surprise, that works perfectly fine to determine what is currently trending in the community

https://hn.moritz.pm

Re: Netlify just sent me a $104k bill for a simple static site

#722

Post by mathgradthrow »

jotaen wrote: post #520
bobfunk wrote: post #317

Netlify CEO here. Our support team has reached out to the user from the thread to let them know they're not getting charged for this. It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact. Apologies that this didn't come through in the initial support reply.

One additional feedback, for consideration: to me, your Pricing page[1] doesn’t make it sufficiently clear that the “Starter” plan may incur costs at all (let alone in this ballpark). It’s now more apparent when looking at it in hindsight, but you have to either read very carefully, or go to the separate “View Features” page to understand this. “0$ to get started, then pay as you go” reads to me: “0$ to get started,…

How on earth could I, as a customer, be sure that netlify hadn't paid someone to DDOS me? If I were in charge of a business like that, I would have that thought constantly...

Re: Netlify just sent me a $104k bill for a simple static site

#723

Post by dns_snek »

nemothekid wrote: post #703
dns_snek wrote: post #669

Earlier quoted context omitted.

They profit when their customers can't hard limit their spend and end up racking up large bills by accident, or for reasons outside of their control. By the way, your comment was flagged which seemed odd, so I looked at your profile and it seems like all of your comments are being (automatically?) flagged and don't appear on HN by default. You might want to talk to the HN staff about that.

1. This doesn't seem like a rational thing to do. A trillion dollar business built on people making mistakes and actually running up a bill? Which exec is getting a bonus when little Johnny gets hit with a $1000 charge on his CS101 project? Doesn't seem likely. 2. To me, it's more likely they don't have one because there are edge cases to consider that make "hard limits" difficult to implement. What is AWS supposed t…

1. Even if you assume perfectly good intentions, they're not incentivized to fix the problem because they're not on the hook for the bill, but stand to profit from it. Their margins are eye-watering, so sending out a $100k bill for a service which cost them no more than $5 to provide doesn't have a downside (other than bad PR, which they seem to be blind to).

If providing this bandwidth cost them $50k rather than $5, I would bet you my entire life savings that they would QUICKLY find a way to add hard limits to their service, no matter what technical challenges they're quoting now.

2. This is probably a 95/5 problem. The vast majority of these sorts of cases that I've seen and read about are caused by increased traffic, which hits the customer either with extortionate egress fees or unintended compute scaling behavior (FaaS/VMs).

Storage is trickier, but you could stop accepting writes or reads after passing some hard limit.

This leaves some edge cases, sure, but it should handle the vast majority of unexpected bills without any destructive actions.

> You can set an alarm in AWS today and the user can decide what to shut off.

That shifts responsibility back on the customer, which is exactly the problem to begin with. I need assurances that I won't be billed more than $X in any given day, or month and this solution doesn't provide that. Maybe their API down, maybe it's serving stale data, or maybe my automation fails for some reason.

> How is all of this communicated?

Hard monthly limits:

Egress: [ $ 10 ] - When exceeded, all outbound traffic is limited to [ 0 Mbps ].

Compute: [ $ 10 ] - When exceeded, scale all compute instances to [ 0 ]

Data Storage: [ 1 TB ] - When exceeded, all writes are rejected

Reads Ops: [ $ 1 ] - When exceeded, all reads ops are rejected

It's really not that hard. Offer these limits on a per-project, or even per-resource level, that would naturally allow you to limit the blast radius. A personal website that has gone viral would likely want to have different limits than an email server under the same account, for example.

Re: Netlify just sent me a $104k bill for a simple static site

#724

Post by belthesar »

chgs wrote: post #289
cotillion wrote: post #153

Earlier quoted context omitted.

My experience is that customers don't really care that much about small amounts of downtime no matter what size you are, people mostly get that unexpected stuff happens as long as you don't get hacked or misplace their data. Customers might complain a bit but seldom leave because of a few hours downtime. This seems to mostly hold true to developers also, GitHub manages to survive just fine after all.

Depends on your service. 20 second downtime on loading HN? Nobody cares. 20 second downtime on the last play of the Super Bowl - big problems. For most internet consumers we’re accustomed to poor service so if a page doesn’t load we’ll assume it’s a local problem and try again 20 seconds later, same with buffering, it’s just something that happens occasionally. This is increasing the case for phone calls too. Legacy…

Sure, but now you're talking about sites with completely different service level objectives, and conversely, different budgets for their hosting. The problem here, to play off of your analogy, is that Netlify is treating every customer, many with SLOs likely less strict than HN as if they are the Super Bowl. This is an assumption that, according to the most recent policy discoverable by looking through their forum posts, is a constraint of their platform, and something they tout as a feature, not a bug.

When users expressed concerns for a similar scenario that the OP experienced on their community forum, Netlify's staff responded with "how likely is this, really?" Only has to happen once to put someone in significant financial harm.

Re: Netlify just sent me a $104k bill for a simple static site

#725

Post by coryrc »

dboreham wrote: post #612
coryrc wrote: post #600

Earlier quoted context omitted.

Traffic doesn't cost money. Bandwidth costs money. Unused bandwidth doesn't cost less than used bandwidth. So, no, you shouldn't pay so much for something that doesn't cost them any money?

> Traffic doesn't cost money. Mostly false. Either transit is billed on a 95th%ile basis (so...more money for more traffic), or if it is flat/netted, you're still paying for the capex for the switch ports (fatter connection to support more traffic means more $$$ for the gear to support it).

At Netlify scale, it'll be the latter. And the appropriate thing to do with free-tier traffic is set lower QoS so it doesn't interfere with paid traffic. This, it doesn't have cost.

You can say "the sum total of free-tier traffic requires X additional connections" but the sudden burst of DoS traffic did not raise marginal costs (besides an inconsequential usage of electricity).

Re: Netlify just sent me a $104k bill for a simple static site

#726

Post by albert180 »

conductr wrote: post #549
noir_lord wrote: post #378

Earlier quoted context omitted.

Any cloud platform should have a spend-stop amount built in. i.e. if I know I average $10 a day, I should be able to put in a "If it hits $50, email me and take it offline". Of course the opposite problem is then people setting that limit too low but since the user defines the limit that's on them not you. This is one of the reasons I still in 2024 rent physical boxes and run the modern stuff on top of them directly,…

> I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped. I still prefer this too. Kinda funny how server resource limitations became a feature and not a bug when it was one of the problems the cloud sought to overcome

Physical Boxes at Hetzner and the likes are usually way cheaper than most cloud offerings

Re: Netlify just sent me a $104k bill for a simple static site

#727

Post by Capricorn2481 »

sph wrote: post #547
ignoramous wrote: post #129

Earlier quoted context omitted.

> Cloudflare's ddos protection Yeah, we got hammered once with over 10TB/mo and noped out of Netlify as fast as we could: https://twitter.com/rethinkdns/status/1370342245841342466 Had to pay the bill in full. Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0. Makes it hard to move to any other platform. As a small tech shop, this is my Hotel California I'm happy to never leave.

> Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0. Makes it hard to move to any other platform. As a small tech shop, this is my Hotel California I'm happy to never leave. Yeah that's how Cloudflare can reach total control over the Internet. With thunderous applause by people that should know better. I know that my position is outright blasphemous in this day and age, where even self-…

I don't understand this take. First of all, moving off of Cloudflare is trivial if you really have an alternative. Second of all, self hosting a static website is easy, but that's not we're talking about here. We're talking about DDoS mitigation, which is not gonna be solved over a weekend hack with a load balancer. At least, not at the scale that matters.

What would the Cloudflare going evil phase even look like? Is it anything like Netlify charging me 100k because they don't provide ANY DDoS protection? I don't see any FOSS tools preventing this problem.

Re: Netlify just sent me a $104k bill for a simple static site

#728

Post by thingification »

mhitza wrote: post #679
thingification wrote: post #639

People deploying low-traffic side projects - to VPSes or similar that allows price capping - and who do use a database: what deployment stack do you use these days? I assume Docker + something is most popular, but what something? Does terraform work sanely for cheap virtual hosting? Ansible? I don't want to manually install any more stuff than the minimum I can get away with!

For my own projects I do Terraform/Pulumi + Ansible. I use Hetzner & DigitalOcean and this setup works great with both. I don't use docker for my projects, as I deploy on RHEL like systems which I'm intimately familiar how to configure (and have snippets I mix and match).

You use both terraform and ansible in the same project? I always thought of them as competitors filling more or less the same role, do you find it useful to use them together? Is it that hetzner and digitalocean TF providers do a good job but provide limited functionality, and ansible fills in the gaps for you?

Re: Netlify just sent me a $104k bill for a simple static site

#729

Post by remram »

bradly wrote: post #698
kej wrote: post #555

Earlier quoted context omitted.

This is something I really like about Nearly Free Speech.net. Their model is that you deposit funds up front, and they will deduct from those funds as you use services. It helps that they actually are nearly free so that a single $20 deposit can last for months or years in many cases. It's bizarre to me that more services don't support billing this way, since there are tons of situations where I would much rather hav…

A couple years ago I switched from a standard, contract-based US cell phone plan to a prepaid service it has felt so much more natural. Have a messed up my autopay and my phone just stop working in the middle of the day? Yep. But You just find some WIFI, pay, and its back on in minutes. I know exactly what my service costs: $35 dollars. No fees, taxes.

That sounds awful and it's not even cheap!

Re: Netlify just sent me a $104k bill for a simple static site

#730

Post by belthesar »

iforgotpassword wrote: post #417
twixfel wrote: post #336

Earlier quoted context omitted.

People don't check for every possible thing that can go wrong, otherwise we wouldn't have time to do anything. I remember when I got charged a "cancelling fee" for cancelling my Adobe subscription that accidentally went past its free trial. The situation was so ludicrous that I had never imagined that they would charge something like 6 months worth of subs me to cancel a monthly subscription. In the end I got out of…

Sorry, maybe this is a generational thing? I always look for a catch when something is "free". Your example as old as time, offer a free trial period but if you don't cancel X days before it ends, you automatically subscribe for another week/month/year. This has extended to getting a big discount for your first year after signup and then you pretty much have to cancel and go to a competitor, or wait until a couple da…

I definitely think this is a generational thing. Like you, I come from the land of "everything costs something" and treat free with suspicion, but also, we have been conditioning people to expect things to be free, and to bury and distract people from the real cost of those things for quite some time.

Two of the largest video content distribution platforms on the Internet, YouTube and Twitch. Free!* (just watch these ads). Store your data on the cloud with Dropbox, Google Drive, or OneDrive! Free!* (just let us harvest data about what you put in there) Hell, 25 years ago, you want to get on the internet? Use NetZero! Free!* (just look at these popups) And this pattern continues to be pervasive. And the real killer here is that we keep getting things for free*, so people that weren't around during the introduction of these tactics have grown accustom to it as if it's how things should be.

So, I agree, we really have a problem here in messaging and in using misleading psychology to bury dark patterns like the true cost of Free in services we use. We probably should be teaching more folks to beware of the true cost of things, that if it's free, you're the product, not the user, so on and so forth.

Post reply on HN