Google Cloud: $72,000 bill overnight • Hacker News Board

Re: Google Cloud: $72,000 bill overnight

#91

Post by wolco2 » Sun, Dec 13, 2020, 1:57 AM UTC

gonzo41 wrote: post #32

ncfausti wrote: post #29

Earlier quoted context omitted.

This was my thought as well. What other kinds of businesses or services let you run up a bill of tens of thousands of dollars and then say, "Ok, you made a mistake, you can take it back"? Any educated guesses on what this compute might actually cost Google? I assume they're able to do this because the fixed costs have mostly been paid for already and the marginal cost of the electricity, system wear, and bandwidth ar…

It would cost them dollars. An IPhone costs < $100 to make. Obviously there's a development cost to it all. it's also easy to suggest that there's probably a feature missing atm :D

It would be like writing off a stamp.

Re: Google Cloud: $72,000 bill overnight

#92

Post by dap » Sun, Dec 13, 2020, 4:52 AM UTC

closeparen wrote: post #41

dap wrote: post #4

I have seen similar stories with AWS. It’s somewhat shocking to me that there’s no way to ask to get cut off above some dollar limit. Is every customer risking unbounded liability?

Billing is traditionally not built into critical paths, but on async or even batch processing of logs from those systems. I doubt any system at Amazon really knows what you’ve spent until after the fact.

I think that’s true, but that’s a reflection of what people considered important when they built it. It’s not the only way to build things.

Re: Google Cloud: $72,000 bill overnight

#93

Post by dap » Sun, Dec 13, 2020, 4:54 AM UTC

imtringued wrote: post #52

dap wrote: post #4

I have seen similar stories with AWS. It’s somewhat shocking to me that there’s no way to ask to get cut off above some dollar limit. Is every customer risking unbounded liability?

Dollar limits don't make sense for companies because you can't predict which parts of your infrastructure would get shutdown first. Hobbyists don't mind if everything is turned off but they would still want to keep the stored data. There would still be an opportunity to be overcharged on storage even with a dollar limit. However, setting usage limits would be a solution for both companies and hobbyists. AWS could the…

Well, I suggested an option. That doesn’t mean companies have to choose it.

But to be honest, it’s somewhat surprising that companies are willing to take on the risk of unbounded financial liability if someone makes a mistake.

Re: Google Cloud: $72,000 bill overnight

#94

Post by gonzo41 » Sun, Dec 13, 2020, 7:33 AM UTC

wdb wrote: post #40

gonzo41 wrote: post #34

Earlier quoted context omitted.

Next time don't ring, write and email. I've worked in financial sales. What's in the contract counts more than anything anyone ever says.

But is an email that says you only pay for the minutes counts as an contract?

It just creates a record. Doesn't have to be enforceable. but it means the person doing support becomes accountable for the things they've said. Chances are there's telephony recordings of your call. So you could escalate looking to hear those recordings. Emails' just easier in this regard.

Re: Google Cloud: $72,000 bill overnight

#95

Post by jobigoud » Sun, Dec 13, 2020, 9:46 AM UTC

motohagiography wrote: post #5

This happened to me with Linkedin advertising, where my budget of a couple hundred dollars got re-charged to my card up to a couple thousand dollars without notifications. They handled all the complaints not in email, but their web based interface, and memory holed their commitment to refunding me the money after multiple back and forths. This isn't a mistake, the design is their business model. While we don't have a…

Always use a virtual card for this. This way you can tell in advance how much can be drawn max and the expiration date. And even if their database is compromised it won't work elsewhere.

Re: Google Cloud: $72,000 bill overnight

#96

Post by zenexer » Mon, Dec 14, 2020, 10:50 AM UTC

CodesInChaos wrote: post #79

zenexer wrote: post #62

Earlier quoted context omitted.

This person used a new IP address for every single request, so that won’t work. And that’s a growing trend.

I'd assume they use 1000 distinct IPs because the system scaled to 1000 instances and presumably many of them came from related IPs. So it makes IP banning considerably harder, but not impossible.

More importantly, any reasonable IP-based approach would have a lot of false positives. What if there’s an RSS service like Feedly running on the same cloud?

Re: Google Cloud: $72,000 bill overnight

#97

Post by CodesInChaos » Tue, Dec 15, 2020, 8:38 AM UTC

rantwasp wrote: post #84

CodesInChaos wrote: post #82

Earlier quoted context omitted.

There are other costs which can scale pretty high. The overpriced outgoing traffic is one. Certain APIs also charge per request or traffic. The S3 object storage is one example, databases like Firebase another. I believe most of the costs the OP incurred came from firebase and not the 1000 Cloud Run instances.

again. the comparison needs to be apples to apples. it's true that outbound traffic does cost (and in theory if you do a lot of traffic), but the budget instances don't have that much bandwidth to start with. Also if you have a huge amount of traffic depending on what you do you might be better off with S3+cloudfront

Isn't outbound cloudfront traffic just as outrageously expensive as outbound EC2 traffic? My impression is that AWS is simply unaffordable when you have large amounts of traffic.

Re: Google Cloud: $72,000 bill overnight

#98

Post by CodesInChaos » Tue, Dec 15, 2020, 8:40 AM UTC

zenexer wrote: post #96

CodesInChaos wrote: post #79

Earlier quoted context omitted.

I'd assume they use 1000 distinct IPs because the system scaled to 1000 instances and presumably many of them came from related IPs. So it makes IP banning considerably harder, but not impossible.

More importantly, any reasonable IP-based approach would have a lot of false positives. What if there’s an RSS service like Feedly running on the same cloud?

Depending on your product, blacklisting all AWS IPs might be acceptable. For example my company has a VPN exit on AWS, which appears to be blacklisted by twitter.