[dead]
€54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
291–300 of 325 posts
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#292Earlier quoted context omitted.
If I budget enough to store 1TB of data for 1 month, then on the first day of the month I store 2TB of data - what should the behaviour be after 15 days?
Nuke the data. It’s gone forever if you didn’t back it up elsewhere. This should be a meaningful risk mitigation that I can employ to avoid having a catastrophic financial disaster. This isn’t a limit I’m setting at some percentage above expected costs, it’s: “I don’t want to take out a HELOC if something goes wrong”
Unfortunately, a lot of people keep their backups in the same cloud account as their primary data. Thinking that multiple copies and multiple availability zones are sufficient.
For these users, the article’s €54k bill would be replaced with their business data getting wiped out.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#293From the response from the Gemini product lead:
> We are moving to disable the usage of unrestricted API keys in the Gemini API, should have more updates there soon.
It's unacceptable the contract for client-side keys was broken in this manner, and doubly bad that it's taken so long for Google to remediate this issue. The Gemini team needs to publish a postmortem to explain what broke down in the engineering process to allow this to happen.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#294Earlier quoted context omitted.
there is no way to cap your billing on gcp. you can get notifications but that's it. i don't want to get throttled below my quota but some type of spend limit would be good.
Is there a cloud provider that does have hard unbreakable billing caps? Everything I've seen has always been notifications or soft caps. Not talking about fixed-access things like a Hetzer box.
Bunny.net purports to have a pay-as-you-go prepaid credit system that sounds like it works the way people want, and with their description of the way it works probably being sufficient to be legally enforceable if it turns out that it actually works differently and you were to end up with a surprise bill from them. And evidently it really does work that way; see this post from a couple weeks ago: https://news.ycombinator.com/item?id=47676416>
The only other provider known to work that way is NearlyFreeSpeech.NET, which serves a completely different market segment (so much so that it might as well not even be considered the same kind of product/service).
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#295Earlier quoted context omitted.
This should be illegal. If a contractor your hired to swap out a tile on your bathroom floor billed you for remodelling your back garden, you would obviously have the legal right to refuse that.
"We can either charge per tile, per job or on demand. Or you can have us on call for a year and get any of the former at a discounted rate." "Per tile. Lay tiles until I say stop" >you fall asleep "Wtf why are you still laying tile" "You said per tile and lay until you say stop. That'll be 50k please" How is this the contractors fault?
"Can you lay tiles until I say stop, or until it's about $250 worth, whichever comes first"
"No, as one of the top tile layers in the country I can't do that, for your own protection. What if fifty elephants came and wanted to use your bathroom all at once? You'd feel pretty dumb having to reject them instead of me simply automatically adding $1 million to your bill"
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#296Earlier quoted context omitted.
The employer is held liable in such a scenario.
Sounds right. Not sure if this is the position: If you’re coding, you should pay for your mistakes, if you’re driving a forklift (sober/responsibly), your employer should pay?
If you are coding your employer pays for it too. If I take the site down and we lose $5 million I am not personally liable for that.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#297Earlier quoted context omitted.
If that happens, you create a support ticket and AWS/GCP/Azure wave it, especially the first time. They're aware that billing per usage can have surprise effects, but at the same time they don't want to kill their customers' workloads and delete their data, so it is what it is.
It's quite easy to check responses to other customers in other threads there, and somehow I see quite a lot of "oh, go to that other support" and ghosting. If you create support ticket on hacker news, then yes, you will probably get it waved. It's somewhat sad that HN is their support forum now.
Send me a PDF of your bill, and I will happily print out 10 copies so I can wave them all above my head
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#298Earlier quoted context omitted.
If I set a limit, and you cut off my service because I reached the limit, I would definitely not "complain just as much" as if I set a limit and you allowed me to spend past it. We're not talking about an EC2 or EBS volume here, this is access to an API.
Meh, you probably would complain. Maybe you forgot you set it. Now your project is taking off, making money, and it got nuked. Why aren't we talking about an EC2 - is that not a cloud compute service? People have been complaining about cloud billing since long before LLMs. Anything to say about the technical problem of constantly monitoring many services against a project or account-level limit?
Why aren't we talking about an EC2? Because this is a thread about the Gemini inference API. Loss of service would be restored on payment, not permanent. But that's besides the point: I as the customer set a limit, and you as as the service provider did not adhere to it.
I've worked on a number of systems and while it is sometimes impossible to stop at an exact limit, I am confident that it is feasible to stop with less slippage than occurred in this scenario. And at the companies I've worked at, within a margin of error that we're able to absorb any slippage ourselves, as these losses are made up elsewhere, and are worth the customer goodwill. If we can do it, I'm sure Google can.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#299> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…
That's actually crazy. So I can build a project I love, that does good, but somehow get in a situation where I'm accidentally paying 30.000€ (or 50.000€) to a big tech company? How is that fair? I mean yes, as a software engineer, you ought to reflect on all possible weaknesses, but there was a time when overlooking something meant something completely different than being down 30/50k. That is actually life-altering.
agree. the real problem isn't that hard caps are "technically impossible" — it's that the incentive to build them is backwards. a hard cap that stops a runaway process costs the cloud provider money. a "budget alert" that fires after the fact costs the customer money.
the 10-minute delay in billing processing is doing a lot of work in that logankilpatrick comment. at $4k/minute burn rates, that's still a $40k exposure window
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#300Earlier quoted context omitted.
Which cloud provider actually prioritises features that cut off your money supply? Because AWS sure as shit doesn't either.
Amazon, Microsoft and Google don't offer hard cap. Most other/smaller public cloud providers do. The reasons are quite obvious.
From the linked thread: https://ai.google.dev/gemini-api/docs/billing#tier-spend-cap...
The warnings firing off hours later is obviously awful design, but the warnings are just warnings. The spend caps are something different and Gemini has them at the very least.
For most use cases where businesses use the cloud hard spending caps are an awful idea anyway. Killing your servers the moment you start picking up loads of new customers is a surefire way to kill your big growth opportunity at exactly the wrong time.
Of course, if you're not planning for sudden massive growth, you'd be crazy to host your stuff with the big three cloud providers.