The scarier version of this problem is coming. Imagine AI agents making API calls across multiple vendors. Each vendor tracks usage in isolation. But no one knows what the agent is spending across all vendors. Nobody building the cross-vendor spending cap today.
€54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
301–310 of 325 posts
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#302Earlier quoted context omitted.
"We can either charge per tile, per job or on demand. Or you can have us on call for a year and get any of the former at a discounted rate." "Per tile. Lay tiles until I say stop" >you fall asleep "Wtf why are you still laying tile" "You said per tile and lay until you say stop. That'll be 50k please" How is this the contractors fault?
"Can you lay tiles until I say stop, or until it's about $250 worth, whichever comes first" "No, as one of the top tile layers in the country I can't do that, for your own protection. What if fifty elephants came and wanted to use your bathroom all at once? You'd feel pretty dumb having to reject them instead of me simply automatically adding $1 million to your bill"
But in this example, the last line of your story is the customer going “yeah, sounds good, let’s do it and hope that doesn’t happen” and signing the agreement.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#303Two things that should be default on any GCP project touching generative-AI APIs: 1 API-key restrictions by HTTP referrer AND by API (`generativelanguage.googleapis.com` only), 2 a billing budget with a Pub/Sub "cap" action, not just an email alert. Neither is on by default, and almost nobody sets them before shipping. 13 hours is actually fast for detection. most teams find out at end-of-month reconciliation.
I want API keys with monthly and hourly quotas and RATE LIMITING. like 50k requests per hour, above that 1/s/client up to 20 req/sec. I don't want to shotgun my service for every user if one user is misbehaving. I want to set rate of bleeding
[deleted]
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#304As others have said, this is a "feature" for Google, not a bug. There is no easy way to set a hard cap on billing on a project. I spent the better time of an hour trying to find it in the billing settings in GCP, only to land on reddit and figuring out that you could set a budget alert to trigger a Pub/Sub message, which triggers a Cloud Function to disable billing for the project. Insanity.
My favorite Google LLM benchmark is asking Gemini models to create a script that fetches API usage (just request counts) for a project from GCP. 100% failure rate.
I've yet to receive an accurate response from Gemini about GCP services, beyond completely trivial topics. The most recent, I think, was Gemini advising me that I could attach an existing pd SSD PVC to a n4 or c4 VM. For whatever unknowable reason, Google doesn't allow this and doesn't offer a migration path, and Gemini doesn't "know" anything about it either. It's wild.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#305Earlier quoted context omitted.
Almost wish the people defending it were paid. Almost more intelligent to rush to the defense if there were a direct financial benefit. Part of it is possibly the curse of knowledge. Someone in the 99th percentile of cloud configuration experts simply can't recall their junior dev days.
In my junior dev days I always paid for the resources I used. Just because you consume a lot of resources by accident that doesn't mean you shouldn't have to pay for it. Accidents do not absolve you from liability.
This is not about paying or not paying. It's about cloud providers not having working tools that let you limit your spending.
If I don't set up a budget and run up a huge bill, fine, sure, I should probably pay for it. But if I follow best practices and set up a rule like: "if usage > X €, then stop accepting jobs", and I do it correctly according to the vendor's instructions, yet it still lets me blow past the budget, that's entirely on the vendor.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#306Back in 2020–21, we were teaching our students how to stand up and configure cloud services, and I decided that a good extension of my homelab would be in AWS, so that I could learn basic cloud administration.
I was using the Free Tier for starters, of course. I managed to start up a working MediaWiki server on a Linux machine in EC2. I began to explore some of the more esoteric options and always checked out the extra-secure methods, and IAM and so forth.
My MediaWiki had a lot of spammers registering accounts. They weren't actually able to make edits, but I couldn't seem to stop them from creating user accounts. And I always felt rather... naked in terms of securing the Linux system itself. It seemed like the entire Internet had an Eye of Sauron focused on my open TCP ports and they were port-scanning it and running pentests 24/7. I honestly couldn't keep up!
Ultimately, I did realize that I could never constrain the budget to an affordable $20 or $30. Signs seemed to indicate that any malicious traffic could crank up my network egress costs alone! There were some rudimentary controls but they would never permit a full-scale shutdown of all services that could actually cost money.
So I shut down the cloud services and abandoned my Amazon AWS account. Migration to the cloud might seem like a good value proposition for any business that can't handle its own machine rooms or its own I.T. team to manage physical infrastructure. But it's an unconstrained cost nightmare waiting to happen for basically anyone of any scale. I would never recommend it, for personal or business use, until that aspect is somehow brought under control.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#307Earlier quoted context omitted.
Sounds right. Not sure if this is the position: If you’re coding, you should pay for your mistakes, if you’re driving a forklift (sober/responsibly), your employer should pay?
If you are coding your employer pays for it too. If I take the site down and we lose $5 million I am not personally liable for that.
That mutually exclusive with our originating comment you think?:
"...I always paid for the resources I used. Just because you consume a lot of resources by accident that doesn't mean you shouldn't have to pay for it."
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#308Earlier quoted context omitted.
Almost wish the people defending it were paid. Almost more intelligent to rush to the defense if there were a direct financial benefit. Part of it is possibly the curse of knowledge. Someone in the 99th percentile of cloud configuration experts simply can't recall their junior dev days.
In my junior dev days I always paid for the resources I used. Just because you consume a lot of resources by accident that doesn't mean you shouldn't have to pay for it. Accidents do not absolve you from liability.
It's not about not paying for the resources you use. It's about not having any mechanism to limit those resources, despite that being an entirely reasonable thing for the cloud providers to provide.
Using these platforms is like giving everyone in your business a credit card with an infinite limit. If someone steals it, or anyone makes a mistake, your liability is literally unlimited for no reason at all other than complete laziness by the counterparty.
These are completely normal and expected concepts in commercial contracts that the cloud providers just have no respect to provide. I would even wager that their bigger customers have this in their contracts and only SMBs get screwed like this.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#309Earlier quoted context omitted.
See also: Why is the default cap so low? I lost €78bojillion because my API stopped working.
Monitoring could pick this up in minutes rather than how long this took to discover
Monitoring could have detected overspend as well.
My point is either choice, caps or no caps, has its cons.
Re: €54k spike in 13h from unrestricted Firebase browser key accessing Gemini APIs
#310Earlier quoted context omitted.
We have spend caps at the billing account level and the project level (developer set) in the Gemini API now. There is up to a 10 minute delay in processing everything but this should significantly mitigate the risk here: https://ai.google.dev/gemini-api/docs/billing#tier-spend-cap... By default, new Tier 1 paid accounts can only spend $250 in a given month.
I just find it extraordinary that the biggest tech company in the world can do cutting edge real time AI for millions of people, run Youtube and of course all the other google services with having literally the smartest people in the world and unlimited resources on board, but still can't keep real time track of the user's current billing and their spending limits, it's all best effort still. Somehow it doesn't add u…
Not just Google, also Microsoft and Amazon. Real-time cost tracking is technically impossible to solve according to the major cloud providers. I have huge respect for those sales & finops engineers.