Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

€54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

discuss.ai.google.dev

Post reply on HN
Print view View original

221–230 of 325 posts

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#221

Post by nurettin »

bartread wrote: post #190
nurettin wrote: post #176

Earlier quoted context omitted.

> Trading platforms can guarantee a maximum slippage on stops Yeah no, physically impossible. If nobody is selling at that price, there is no guarantee your sell stop will execute near that price. They can sweep the market, find the best seller price and execute. There might be a costly way to do it with microservices as I indicated, but your example easily falls apart.

Not impossible to do: they can hedge and/or absorb the cost, hence the premium. They usually also specify a (fairly large) minimum distance for such stops.

That's exactly what I proposed in my response. Big corp can waiver the extra costs to match your limit. Glad we finally got to that part of my response. The question is: will they? Probably not. Do brokers do it? I haven't seen any. Maybe you know more.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#222

Post by il-b »

lxgr wrote: post #111
singpolyma3 wrote: post #39

Earlier quoted context omitted.

Um. What? In what world are API keys not secrets?

Public API keys are a thing. Arguably they are poorly named (it's really more of a client identifier), and modeling them as primarily a key instead of primarily as a non-secret identifier can go very wrong, as evidenced here.

Yeah, just like “public key” in the cryptography sense

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#223

Post by harrouet »

benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

As a manager I avoid Google Cloud for this kind of customer-service disasters; but as someone who has dealt with large-scale billing systems in the telecom world, probably similar to that of Google Cloud, I am not surprised that it takes 10 minutes to consolidate all the usage logs of a customer for billing.

For telephony, it sometimes takes days when roaming is involved.

You have to imagine TB/sec of data, if not more, coming from thousand of potential sources, and queuing for aggregation to the proper company account, all having to be auditable. This is not a small engineering feat and it can't be real-time.

With that said, telcos usually include in their business model around 2-3% of bad debt (i.e. revenue that won't get paid), which accounts for frauds like this one. Given that the customer seems in good faith and has taken measures upon being notified, Google should manage this bill shock a bit more elegantly.

Moreover, the fact that this happened immediately after this key opened the AI gates means that pirates permanently scan for the permissions of all the keys they could gathers. Google could and should detect that and act upon it.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#224

Post by sofixa »

benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

> So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.)

Yes, it's technically+business impossible. To implement a hard cap, a bill never to go over, they'd have to cut your service, but also delete all your data in databases, object storage, data lake, etc. This is simply not an option, so they take the different option of authorising support to wave surprise surcharges / billing DDoSes.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#225

Post by sofixa »

Leomuck wrote: post #145
benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

That's actually crazy. So I can build a project I love, that does good, but somehow get in a situation where I'm accidentally paying 30.000€ (or 50.000€) to a big tech company? How is that fair? I mean yes, as a software engineer, you ought to reflect on all possible weaknesses, but there was a time when overlooking something meant something completely different than being down 30/50k. That is actually life-altering.

If that happens, you create a support ticket and AWS/GCP/Azure wave it, especially the first time. They're aware that billing per usage can have surprise effects, but at the same time they don't want to kill their customers' workloads and delete their data, so it is what it is.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#226

Post by mexicocitinluez »

Nathanba wrote: post #144
Illniyar wrote: post #75

I think the logistics of calculating cost in real time is something that is extremely hard. I don't think there is one big cloud service provider that has hard limits instead of alerts. As long as they revert the charge when notified of scenarios like this , and they have historically done so for many cases, it's fine. It's an acceptable workaround for a hard problem and the cost of doing business ( just like Credit…

Why would it be hard to calculate cost? Multiply a fixed price * requests/time ? It doesn't have to be exact in real time, it just has to report something approximately useful in realtime. It's absolutely not fine to be at the mercy of other people, that's what we buy cloud products or really any products for: So that we are not at the mercy of hardware faults, bad weather, bad teeth, hunger, thirst, [insert anything…

I'm guessing the answer is simply money. It's less expensive to deal with people like this this than it probably was to prevent it. Right now, they seem to run very sparsely, so ramp that up (if it's every 3 hours and they want to change to 5 minutes that's like a 6000% increase) and they're probably paying more than it costs to employ people to return credits or fears of people leaving.

It sucks, but that's unfortunately the world we live in until something changes.

The US could rely on an agency like the CFPB to prevent this, but that was gutted under the current admin.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#227

Post by hilariously »

clapthewind wrote: post #85
hilariously wrote: post #35

Earlier quoted context omitted.

This reads like 100% an LLM comment. by design -- the enforcement Nothing new here - what is new is the A thing before anyone noticed - another thing, billing in hours, damage in minutes. has the signal, doesn't expose the control Every one of those "exposes the signal" to me.

Do not get hung up? Sounds like English ESL rewrote some insights for language. Content > Form.

At this point its much more polite to write badly than use an LLM to rewrite your content. The form tells me you do not care to interact with me in a genuine way.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#228

Post by benterix »

sofixa wrote: post #225
Leomuck wrote: post #145

Earlier quoted context omitted.

That's actually crazy. So I can build a project I love, that does good, but somehow get in a situation where I'm accidentally paying 30.000€ (or 50.000€) to a big tech company? How is that fair? I mean yes, as a software engineer, you ought to reflect on all possible weaknesses, but there was a time when overlooking something meant something completely different than being down 30/50k. That is actually life-altering.

If that happens, you create a support ticket and AWS/GCP/Azure wave it, especially the first time. They're aware that billing per usage can have surprise effects, but at the same time they don't want to kill their customers' workloads and delete their data, so it is what it is.

Here, I corrected that for you:

> you create a support ticket and spend sleepless night praying that AWS/GCP/Azure wave it

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#229

Post by plorkyeran »

sofixa wrote: post #224
benterix wrote: post #90

> We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours > By the time we reacted, costs were already around €28,000 > The final amount settled at €54,000+ due to delayed cost reporting So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "ha…

> So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.) Yes, it's technically+business impossible. To implement a hard cap, a bill never to go over, they'd have to cut your service, but also delete all your data in databases, object storage, data la…

You can have a hard cap on compute spend while letting storage go over. Surprise huge bills are approximately never due to storage.

Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs

#230

Post by theanonymousone »

adriand wrote: post #167
janandonly wrote: post #138

Earlier quoted context omitted.

Yet another good reason to use a pre-paid service. There are many to choose from now, like Openrouter.com, PPQ.ai, and routstr.com.

You mean openrouter.ai. And yes, on reading this blog post, I immediately reviewed my API keys in OpenRouter to make sure that they were capped. My prod key was capped at $20/day (phew!) but my dev key had no cap, which I just updated. What a horrible story.

But isn't OpenRouter anyway prepaid, meaning the most you lose is your current credit?
Post reply on HN