Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud: $72,000 bill overnight

theregister.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: Google Cloud: $72,000 bill overnight

#81

Post by CodesInChaos »

jamesrr39 wrote: post #73
corytheboyd wrote: post #49

Earlier quoted context omitted.

I don’t have any skin in the game here, but what if you don’t build auto-scaling, which keeps this comparison fair. How does pricing differ now?

The difference would be that for a prepaid service, you can build it in, knowing that once it eats through enough $, that's it; there is no more credit to take, power off the service. Whereas the billing by credit card, it will keep on going, and you can end up with these huge bills to pay. But to answer your question (about AWS EC2 vs a DO droplet?), about other costs, you still have data transfer costs, which is cu…

Inbound is free on AWS, outbound starts at $90/TB and goes down to $50/TB eventually.

Compare this to Hetzner where it costs EUR 1/TB and many server types include unlimited traffic at 1 Gbit/s.

I think the $0.01/GB you cited is region internal traffic which was sent through public IPs instead of private IPs.

Re: Google Cloud: $72,000 bill overnight

#82

Post by CodesInChaos »

corytheboyd wrote: post #49
jamesrr39 wrote: post #43

Earlier quoted context omitted.

Sure, you know the price with renting one machine, and if what your doing is not a web app. But what about when you get way more network traffic than your app expected (I've seen HN submissions with exactly this)? And what if you built in some kind of scaling, automatically renting extra machines when you get traffic spikes? They have your card, you pay the $$$.

I don’t have any skin in the game here, but what if you don’t build auto-scaling, which keeps this comparison fair. How does pricing differ now?

There are other costs which can scale pretty high. The overpriced outgoing traffic is one. Certain APIs also charge per request or traffic. The S3 object storage is one example, databases like Firebase another. I believe most of the costs the OP incurred came from firebase and not the 1000 Cloud Run instances.

Re: Google Cloud: $72,000 bill overnight

#83

Post by rantwasp »

hannofcart wrote: post #74
jamesrr39 wrote: post #43

Earlier quoted context omitted.

Sure, you know the price with renting one machine, and if what your doing is not a web app. But what about when you get way more network traffic than your app expected (I've seen HN submissions with exactly this)? And what if you built in some kind of scaling, automatically renting extra machines when you get traffic spikes? They have your card, you pay the $$$.

AWS services such as ECS Fargate let you specify an upper bound to the number of instances that can possibly be spawned. There is even a default 'task limit' they enforce which we had to increase by sending an email request. Typically though, as in the article above, it's the database service scaling that causes big shocks. Rule of thumb would be to always ask if there is an upper bound to every cloud service one use…

you can have auto scaling limits on ASGs on EC2. again, the technical solutions are there. it's a matter of learning how to use it properly

Re: Google Cloud: $72,000 bill overnight

#84

Post by rantwasp »

CodesInChaos wrote: post #82
corytheboyd wrote: post #49

Earlier quoted context omitted.

I don’t have any skin in the game here, but what if you don’t build auto-scaling, which keeps this comparison fair. How does pricing differ now?

There are other costs which can scale pretty high. The overpriced outgoing traffic is one. Certain APIs also charge per request or traffic. The S3 object storage is one example, databases like Firebase another. I believe most of the costs the OP incurred came from firebase and not the 1000 Cloud Run instances.

again. the comparison needs to be apples to apples.

it's true that outbound traffic does cost (and in theory if you do a lot of traffic), but the budget instances don't have that much bandwidth to start with. Also if you have a huge amount of traffic depending on what you do you might be better off with S3+cloudfront

Re: Google Cloud: $72,000 bill overnight

#85

Post by rantwasp »

jamesrr39 wrote: post #73
corytheboyd wrote: post #49

Earlier quoted context omitted.

I don’t have any skin in the game here, but what if you don’t build auto-scaling, which keeps this comparison fair. How does pricing differ now?

The difference would be that for a prepaid service, you can build it in, knowing that once it eats through enough $, that's it; there is no more credit to take, power off the service. Whereas the billing by credit card, it will keep on going, and you can end up with these huge bills to pay. But to answer your question (about AWS EC2 vs a DO droplet?), about other costs, you still have data transfer costs, which is cu…

yeah no. inbound is free on AWS.

Re: Google Cloud: $72,000 bill overnight

#86

Post by Silhouette »

zenexer wrote: post #36

> The ex-Googler reflected that he missed the possibility of pages that link back to each other, causing "infinite recursion." Although tangential to the billing issue, this is reckless. If you’re building a crawler of any kind, please, please, please prioritize ensuring this doesn’t happen so I don’t have to wake up at 3 AM. I run the infrastructure for a moderate-sized site with probably about a hundred million pag…

Which ones are actual attacks, and which are just poorly-made crawlers?

If it walks like an attack and it quacks like an attack...

I am lacking in sympathy for the perp here, as being careless like this has probably caused problems and possibly cost significant money for a lot of people.

However, this is also a compelling demonstration of why cloud services should be required to provide a hard price cap option for safety reasons. I've heard all the self-serving arguments they make about how turning things off surprisingly might be unwanted behaviour and so on. If that's the case, the admin won't set a cap. But there are exactly zero circumstances under which someone who intended to cap their usage at a level that would cost single digits of dollars or remain within a free plan intends or wants to run something that costs four orders of magnitude more than that, and IMNSHO such predatory pricing models should be illegal (assuming that the charges aren't already considered unenforceable by courts under such circumstances; I haven't checked).

Re: Google Cloud: $72,000 bill overnight

#87

Post by zenexer »

hansvm wrote: post #78
zenexer wrote: post #65

Earlier quoted context omitted.

> Also - on the custom UAS: Unless you're trying to make it easy to get blocked/identified then don't take this advice. I can’t speak for other sites, but we’re pretty good at picking up on crawlers that don’t have a unique UA. The problem is that we’re going to have a hard time differentiating your well-behaved crawler from more malicious crawlers, and you’re going to get caught in the crossfire. > if you can get aw…

(2) Is a great idea I hadn't considered. A surprising number of sites require "browser" user-agents but otherwise have well-defined rate limits, robots.txt files, and everything you'd need to write a respectful crawler. I'm not sure that (4) matters for larger sites? Their rate limits are usually a drop in the bucket compared to the background traffic.

#4 was more to avoid being noticed by someone like me before they’ve had their morning coffee. That being said, if anything does go wrong, and you’ve ramped up slowly, at least it gives autoscaling time to respond.

Generally, though, unless you screw up badly, submit forms, or blend in with a more problematic crawler, nobody’s going to care (or even notice).

Re: Google Cloud: $72,000 bill overnight

#88

Post by raphaelj »

franciscop wrote: post #21
raphaelj wrote: post #15

Someone should make a list of all the providers that support some kind of billing budget. It took me way more time than expected to find a CDN that has a budget limit for my personal project (I'm using Bunny CDN)

I've been looking at BunnyCDN and it looks pretty impressive. Is it as good as their landing pages look? Is it like a CDN+DNS so I can point my nameservers there and then configure the CNAMEs inside? (I'm fairly unfamiliar with DNS in general). I've been pretty happy with Cloudflare, but at some point I added my credit card (silly me) and now I live scared of a DDOS costing me a lot of money.

I've been super happy with BunnyCDN for my personal SaaS. Their UI is super nice, and it has been super easy to setup. Costs are really easy to predict, which is awesome for me.

Re: Google Cloud: $72,000 bill overnight

#89

Post by psanford »

rantwasp wrote: post #38
psanford wrote: post #33

Earlier quoted context omitted.

If I were in AWS/GCP's position, I would prefer to send alerts rather than turn off services. Shutting off services can mean destroying the customer's data with no way for them to recover it. That could be from terminated ephemeral disks, or a terminated database, or cutting off a critical upload stream into their instances. Its a lot easier to reduce/forgive a bill when a customer makes a mistake than to recover the…

you can get billing alerts today. it’s not the lack of control mechanisms that prevented this failure. it was the noobism of the user

The over spend was caused by an application bug, not because the user was a "noob".

Re: Google Cloud: $72,000 bill overnight

#90

Post by wolco2 »

rantwasp wrote: post #37
jamesrr39 wrote: post #30

#1 reason why I have never, and never plan to (while it remains this way), use GCP/AWS etc while it would be on my personal bank card. Instead I use DigitalOcean where you have droplet limits that you can set, and the ability the pre-pay if you pay by PayPal, and never enter my bank card. If anyone from DO (or another provider) is reading this, any chance of pre-payment from bank cards? After reading enough of these…

that’s a non-argument. you can do the same with an EC2 instance. you know exactly what it’s gonna cost. it’s this fancy services with “elastic” pricing models that usually get you

I tried to signup with a gift card and it was rejected.

I can attach a gift card to paypal and use it.

Post reply on HN