A startup’s Firebase bill suddenly increased from $25 to $1750 per month • Hacker News Board

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#361

Post by jahewson » Wed, May 17, 2017, 3:59 PM UTC

apatters wrote: post #24

Wowow! In this position my first step would be to determine if there's any legal basis for disputing the charge (an informal conversation with a lawyer or even just someone who's been through a similar experience is a fine start). Even when a billing change is disclosed in advance and the customer actually uses the bandwidth, massive spikes in metered pricing are a tricky area and have attracted the attention of regu…

> telecoms have come under fire by charging sky-high overage fees for mobile data

Telecoms is a very heavily regulated industry.

> Also, immediately dispute the charge with your credit card company

No, don't do that. Most credit card companies require you to try and resolve the issue with the merchant before filing a chargeback. Sometimes they give you a time period, e.g. at least 30 days.

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#362

Post by lucb1e » Wed, May 17, 2017, 4:00 PM UTC

korzun wrote: post #351

apatters wrote: post #24

Wowow! In this position my first step would be to determine if there's any legal basis for disputing the charge (an informal conversation with a lawyer or even just someone who's been through a similar experience is a fine start). Even when a billing change is disclosed in advance and the customer actually uses the bandwidth, massive spikes in metered pricing are a tricky area and have attracted the attention of regu…

> In this position my first step would be to determine if there's any legal basis for disputing the charge Let's calm down. The first step is to pay the bill, set up better infrastructure monitoring on your end, so you don't act surprised that you are consuming 7000% more resources than third party service is reporting. Then, if you want to burn some money, you can file a petition and tell the court that you deserve…

> third party service is reporting

As far as I understood, Firebase didn't report it before. As mentioned, the meter "suddenly jumped" to 100GB and then 180GB (if I remember the numbers correctly).

The author decided to figure it out with support later, which makes sense if the previous number was something like 20-30GB and you guess it's simply in error.

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#363

Post by metaphorm » Wed, May 17, 2017, 4:01 PM UTC

noja wrote: post #356

YCode wrote: post #295

Earlier quoted context omitted.

They don't seem to understand that once you lose trust and reputation in a community you will struggle to get it back. For example this thread has more or less convinced me to take Firebase off the table for any future projects I might have used it for.

Yet they keep doing it, so are their numbers showing otherwise?

not all business decisions are well-grounded in financial upside. a lot of dysfunctional outcomes are a result of organizational dysfunction and broken culture. a company can be making bad decisions for reasons related to internal dysfunction, and be losing money on it, and persist in this because of their dysfunctional culture.

as many in this thread have pointed out: Google has a dysfunctional culture w.r.t. customer service.

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#364

Post by pranavsinghca » Wed, May 17, 2017, 4:01 PM UTC

delhanty wrote: post #164

shaqbert wrote: post #7

Pricing mistake 101: You never ever change those old plans. Instead, you grandfather them. Especially so if you're still a young platform and have most growth ahead of you, the "loss" of not charging the new - and supposedly higher - pricing is gonna be trivial 2 years from now, with lots of new projects coming in. But the NPS hit from pissing your most loyal users off and the subsequent damage to your growth curve a…

Tell that to FastMail, who have pissed me off by deciding to end-of-life the "lifetime" 16Mb member account I set up for my father with a one-time $15 payment. 16Mb is modulo zero these days, but it's enough for him - he just deletes some emails when it gets full. He also gets imap access and FastMail's spam filtering, which is really very good. I can also assign him an email address from a domain that I own that is…

I don't think this should have ever been offered.

Curious - how could FastMail ever properly close/sunset these accounts? Were they going to regularly check on all of their customers to see if they had passed away? Was there some 'we close the email if you don't log in for 5 years straight'?

Serious question - how should these type of lifetime promises ever be properly managed? (especially when the customer is an individual)

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#365

Post by ocdtrekkie » Wed, May 17, 2017, 4:01 PM UTC

brianwawok wrote: post #341

ocdtrekkie wrote: post #216

Earlier quoted context omitted.

The idea of having to pay extra to talk to them about their screw up is pretty laughable.

Why? I like the fact I can pay one price for self support and another for 15 minute resolution. Why force people to pay for support if they don't need it?

I think there's a major difference between say, support figuring out how to implement their product, which is a you problem, their service is working fine.

And a wholly different one if they screw up and you can't talk to them about it without paying them.

The former is an incentive to self-serve and investigate yourself, the latter is an incentive for the company to provide shoddy service to people pay to resolve it.

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#366

Post by mayop100 » Wed, May 17, 2017, 4:04 PM UTC

[Firebase Founder here] I’m very sorry for the surprise and frustration experienced by the poster, especially due to problems working with Firebase support. We’re embarrassed by the level of communication on our side, and we’ll be working directly with this developer to resolve the issue.

There are a couple of things I’d like to clarify for the group, to help folks understand what happened here, and hopefully help others avoid the same problem.

1 - The Firebase Realtime Database charges for SSL overhead on all requests (we charge for all OSI Layer 5 network usage). We’ve always had a policy of charging this way. Unfortunately, we introduced a bug late last year that began undercharging for SSL, so when the bug was fixed it surprised many people who had gotten used to the lower numbers. For most people, the change was very small. For a small number of people though with exceptional use cases (ie. tons of small network requests from IoT devices without support for session tickets), this can result in a larger change. We identified and contacted developers who were significantly affected, but this customer didn’t get the email. We should have done better. (we mention this change in our FAQ -- https://firebase.google.com/support/faq/ -- under “Why was my Realtime Database reported bandwidth lower than average”)

2 - We recently started actually enforcing overages on legacy plans and our current fixed-price ($25/mo Flame) plan. This is new -- in the past, we allowed unlimited usage on every plan, since we hadn’t built the tools to control this. This meant that many of our developers were getting far more than the listed limits for free. So you may start receiving emails now warning you of bandwidth overages, not because your usage patterns changed, but because we’re now enforcing limits for your existing usage. If you upgrade to the Blaze plan, you’ll start being billed for your full usage amounts -- so double-check your database’s “Usage” tab before upgrading.

I’ll be looking into our profiling tool to see if we can improve it to give a more complete picture of costs.

Again -- I want to apologize to the poster for the poor support experience. If others on the thread have had similar problems and feel they are not getting the attention they want from support, feel free to email me directly as well: andrew@firebase.com

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#367

Post by _jal » Wed, May 17, 2017, 4:04 PM UTC

corford wrote: post #105

flipp3r wrote: post #14

Reminds me of getting Bait-and-Switch'd by Google App Engine in 2011. From 10's of euros to 1000's. > They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options. Sounds like Google did a great job with Firebase. Definitely gives you that Google-feel of unresponsi…

I learnt this lesson the hard way. Solution: get a separate debit (not credit!) card for all cloud stuff. Make sure you only transfer enough money on to it each month to cover what you reasonably expect your bills to be (and that you can quickly top it up in hours if you legitimately need to). Worst case, if AWS or Google decide to fuck you over, let the bill bounce. This way you've still got funds on hand to deal wi…

This is excellent advice. In general, I've personally imported lessons about accounting controls I learned running a company to personal finance. Yes, it is a pain.

But Paypal, for instance, can only empty a bank account that is used for nothing else, any service like this (open-ended liabilities with shitty/no support) is isolated to debit cards, etc. Nobody gets permission to charge open-ended amounts to credit cards, and nobody gets automation rights to the "real" bank accounts, period.

There's a business model here somewhere for someone to automate all of this. It involves substantially more time to keep organized than my prior two-banks/two-credit cards model. Personal finance apps just aren't designed to treat, for instance, debit cards as transient.

As others note, sure, services that do something like this can still come after you. But who has the money in their pocket right now matters a lot when resolving disputes like this.

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#368

Post by lucb1e » Wed, May 17, 2017, 4:05 PM UTC

jamesk_au wrote: post #167

Possibly relevant (from the Firebase FAQ[1]): Why was my Realtime Database reported bandwidth lower than average between September 2016 and March 2017? For our bandwidth calculations, we normally include SSL encryption overhead (based on layer 5 of the OSI model). However, in September 2016, we introduced a bug that caused our bandwidth reporting to ignore encryption overhead. This might have resulted in artificially…

How is expended computational power (to encrypt) relevant for the bandwidth bill though? I understand it's an expense for them, but (1) 7500% is not a correction in measurements (they must have noticed that over 2 years if it actually costs that much) and (2) it's a completely new expense category (if your plans are bandwidth-based, you can't just force someone into a larger plan when bandwidth stayed the same, you should announce and explain the change in plans).

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#369

Post by steveklabnik » Wed, May 17, 2017, 4:08 PM UTC

danellis wrote: post #237

specialp wrote: post #214

Earlier quoted context omitted.

Now in the USA as of 2012, banks cannot turn on overdraft protection on a debit card unless the customer requests. Prior to this they were taking your day's debits and applying the largest one first. So suppose you bought lunch for $10 and a coffee for $3 then your cable company later in the day billed you for an amount in excess of your account. They would then apply the cable company bill first, then the 2 smaller…

The order wouldn't matter in that case, would it? Either way it's still deducting the sum of those three transactions at the end of the day. I thought what they were doing is applying all your debits, charging you if that puts you into overdraft, and then applying any credits. If they applied the credits first, you wouldn't go below zero.

I had this happen to me in real life, back as a broke college student. The difference is the number of transactions you get hit with the overdraft fee for.

Consider a bank account with $20 in it, the day before payday, when you forget your World of Warcraft subscription hits today instead of tomorrow. That's $15. But you buy:

  * Lunch: $5
  * Coffee: $2
  * Dinner: $5
  * A late-night snack: $5

Then, the subscription hits at midnight.

If the transactions happened in-order, the subscription causes the overdraft, and you're charged $25 or whatever (yeah they are very high). But in the way banks used to do it, they'd take your total ledger for the day, and order it by highest $ first. So, subscription first, $5 left. Lunch, $0 left. Coffee, dinner, and snack are three transactions when overdrawn, so that's $25 * 3 = $75 in fees instead.

Banks claimed that this process was because you'd probably want your highest-amounts paid first, as it'd be something like rent money. In the end, it means way more fees for them, though.

$75 in fees is especially brutal when you don't have much money in the first place; it pretty much caused me to not be able to even go to 7-11 for a candy bar for two months.

Re: A startup’s Firebase bill suddenly increased from $25 to $1750 per month

#370

Post by camus2 » Wed, May 17, 2017, 4:10 PM UTC

tlarkworthy wrote: post #305

betageek wrote: post #15

The detail that amazed me is the lack of visibility of the issue in reporting tools - it's all very well changing the terms, but not giving your customer any ability to see what is increasing the bill seems massively unfair.

The database usage tab in the web console tells you how much bandwidth you have consumed including SSL. Author's issue was with the commandline profiler. (A Firebase dev)

No the issue is you forcing him to move to the "pay as you go" plan or shutting down his app.