Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

€54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

discuss.ai.google.dev

Post reply on HN
Print view View original

231–240 of 325 posts

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#231

Post by benterix »

sofixa wrote: post #224
benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

> So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.) Yes, it's technically+business impossible. To implement a hard cap, a bill never to go over, they'd have to cut your service, but also delete all your data in databases, object storage, data la…

This argument simply doesn't hold water - their (smaller) competition solved this problem over a decade ago.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#232

Post by Barbing »

charcircuit wrote: post #219
Barbing wrote: post #205

Earlier quoted context omitted.

Almost wish the people defending it were paid. Almost more intelligent to rush to the defense if there were a direct financial benefit. Part of it is possibly the curse of knowledge. Someone in the 99th percentile of cloud configuration experts simply can't recall their junior dev days.

In my junior dev days I always paid for the resources I used. Just because you consume a lot of resources by accident that doesn't mean you shouldn't have to pay for it. Accidents do not absolve you from liability.

Interesting!

I know software is special. That's why software defects are acceptable while a crumbling bridge is not.

With that said, should this apply to other industries? If I clip a warehouse shelf on my first day driving a forklift, should my wages be garnished for life to cover the inventory? Or is the inherent nature of the logistics industry such that an accident does not always imply liability? (Or other)

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#233

Post by sofixa »

ch0wn wrote: post #142
benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

This should be illegal. If a contractor your hired to swap out a tile on your bathroom floor billed you for remodelling your back garden, you would obviously have the legal right to refuse that.

Your analogy is well off.

You hire a contractor and agree they'll bill you per tile, regardless of how many tiles there are. They bill you per tile. End of story.

For a more acurate comparison, consider a utility. You agree to pay for your electic bill. It's not the utility's fault you invited all your friends who decided to run a crypto mining LAN party, and they can't cut you off lightly because it might literally kill you (e.g. you live in a hot place and rely on AC to stay alive).

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#234

Post by aleksiy123 »

benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

For everyone here, you should be able to use quotas in order to set actual limits, you can lower them yourself as a practical solution.

https://docs.cloud.google.com/docs/quotas/view-manage

Quotas are real time or near real time.

Real time spend limits are probably never going to happen. Actual $ amounts are calculated by a centralized billing system offline in batch.

It sounds easy but it’s bonkers complicated, because of things like discounts, free tiers, committed usage, currency conversions and having to support every payment and deal structure in GCP.

Individual eng teams rarely actually think in dollar amounts, they think in the abstraction which is quotas.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#235

Post by nhuser2221 »

benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

Happened to me to, luckily only was 40$, restricted the api the next day. They were using gemini 3 flash which I am not using.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#236

Post by 827a »

tantalor wrote: post #67
827a wrote: post #21

I said this when this finding was originally posted and I'll say it again: This is by far the worst security incident Google has ever had, and that's why they aren't publicly or loudly responding to it. It's deeply embarrassing. They can't fix it without breaking customer workflows. They really, really want it to just go away and six months from now they'll complete their warning period to their enterprise contracts…

What does this have to do with security?

Billing control is security, to be clear, but beyond that: The key permissions that enable anyone to generate text also grant access to all GCP Generative AI endpoints in the project they were provisioned in. That includes things like Files that your system might have uploaded to Gemini for processing, and querying the Gemini context caches for recent Gemini completions your system did. Both of these are likely to contain customer-facing data, if your organization & systems use them.

If you're hearing this and your gut reaction is This can't be real; We're on the same page. Its a staggering issue that Google has categorically failed to respond to. They automatically added this permission to existing keys that they knew their customers were publishing publicly on the internet, because the keys are legitimately supposed to be public for things like client-side Firebase access & Google Maps tile rendering.

They did not notify customers that they were doing this. They did not notify customers after this issue was reported to them months later by Truffle. They did not automatically remove the additional key grants for customers. They continue to push guidance targeted at novices like "just put the Gemini key behind a proxy (that's also publicly exposed on the internet)", which might solve the unintentional files and caching endpoint leaks but doesn't solve the billing issue. They denied that Truffle's initial report was even valid, until Truffle used the Internet Archive to find a Google internal key from 2023, published for a Google Maps widget or something, before Gemini was even released, that was still active, and used it to demonstrate to Google that, hey, anyone can use this key to get Gemini completions on the house, is there anyone driving this ship??" Google fixed the permissions on that specific key. And did nothing else.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#237

Post by croes »

dabedee wrote: post #42

As others have said, this is a "feature" for Google, not a bug. There is no easy way to set a hard cap on billing on a project. I spent the better time of an hour trying to find it in the billing settings in GCP, only to land on reddit and figuring out that you could set a budget alert to trigger a Pub/Sub message, which triggers a Cloud Function to disable billing for the project. Insanity.

> Insanity

You mean cash machine

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#238

Post by Leomuck »

benterix wrote: post #228
sofixa wrote: post #225

Earlier quoted context omitted.

If that happens, you create a support ticket and AWS/GCP/Azure wave it, especially the first time. They're aware that billing per usage can have surprise effects, but at the same time they don't want to kill their customers' workloads and delete their data, so it is what it is.

Here, I corrected that for you: > you create a support ticket and spend sleepless night praying that AWS/GCP/Azure wave it

Exactly! I know, some of those companies sometimes refund you, but if your livelihood depends on it..? That's a crazy situation to be in as a mere developer.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#239

Post by spacebanana7 »

Bridged7756 wrote: post #220
time0ut wrote: post #110

It is scary building on the public cloud as a solo dev or small team. No real safety net, possibly unbounded costs, etc. A large portion of each personal project I do is spent thinking about how to prevent unexpected costs, detect and limit them, and react to them. I used to just chuck everything onto a droplet or VPS, but a lot of the projects I am doing lately need services from Google or AWS. I tend to prefer GCP…

I wonder what happens if you just decide not to pay. Surely that would have some legal implications in the US, but what about elsewhere?

There are very few countries where consumer rights apply to B2B transactions, especially if it’s multiple people operating as a “small team”.

A solo dev however might be able to present themselves as a retail consumer, and leverage some trading standards related rules for unclear pricing or something similar.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#240

Post by bux93 »

juancn wrote: post #191

These are all poorly designed systems from a CX perspective (the billing systems). Billing is usually event driven. Each spending instance (e.g. API call) generates an event. Events go to queues/logs, aggregation is delayed. You get alerts when aggregation happens, which if the aggregation service has a hiccup, can be many hours later (the service SLA and the billing aggregator SLA are different). Even if you have ha…

It's not typically a problem that usage is event driven. At least not for prepaid phone plans. Or debit cards. Or mailboxes. Or any myriad of prepaid or quota'd services. It's not rocket science, just a bad business practice on the part of Google.
Post reply on HN