How we spent $30k in Firebase in less than 72 hours • Hacker News Board

Re: How we spent $30k in Firebase in less than 72 hours

#101

Post by superfrank » Thu, Aug 02, 2018, 4:46 AM UTC

t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

> This is why infinitely scaling pay-as-you-go cloud services terrify me.

It's been a little over a year since I used Firebase in production, so maybe this has changed, but the funny thing is Firebase DB doesn't infinitely scale, despite them advertising that it does.

The Firebase DB caps out at 100k active connections and according to them (at the time) it's a technological limit on their part, so they cannot go higher even if they wanted to.

When we brought this up, they told us they were technically unlimited because you could shard your data into different DBs if you needed more connections, which is like saying all restaurants are all you can eat because you can keep buying more food.

Re: How we spent $30k in Firebase in less than 72 hours

#102

Post by ericpauley » Thu, Aug 02, 2018, 4:54 AM UTC

I've been noticing a steady rise of posts from hackernoon by amateur developers who think they'll be the next great tech blogger. I'm not saying I could do any better, but why are these posts suddenly getting so much attention?

Re: How we spent $30k in Firebase in less than 72 hours

#103

Post by toast0 » Thu, Aug 02, 2018, 4:56 AM UTC

thatjsguy wrote: post #63

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

So I completely agree, but let me play devil's advocate.

If you do go and track down the problem in your depedency and file a bug, one of two things is likely to happen: they close it and say it's fixed in the latest version or they refuse to accept your bug because it's filed against an old version.

Skipping the track it down part and just jumping into upgrading can be a time saver. It works fairly well if you fit into the 'common' part of the user base with frequent updates. (Incidentally dependencies with frequent updates are kind of a pain)

Re: How we spent $30k in Firebase in less than 72 hours

#104

Post by itcmcgrath » Thu, Aug 02, 2018, 4:59 AM UTC

superfrank wrote: post #101

t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

> This is why infinitely scaling pay-as-you-go cloud services terrify me. It's been a little over a year since I used Firebase in production, so maybe this has changed, but the funny thing is Firebase DB doesn't infinitely scale, despite them advertising that it does. The Firebase DB caps out at 100k active connections and according to them (at the time) it's a technological limit on their part, so they cannot go hig…

Note this is about Cloud Firestore, not Firebase RTDB. It's very different infra. We announced our beta limits increased to 1M concurrent connections at the Next 18 conference, and we'll continue to improve from there.

Re: How we spent $30k in Firebase in less than 72 hours

#105

Post by [deleted] » Thu, Aug 02, 2018, 5:09 AM UTC

marktangotango wrote: post #17

everdev wrote: post #4

Earlier quoted context omitted.

Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.

What’s the difference between a cap limiting your traffic and server cpu maxed out, or db connection pool or ... limiting your traffic?

[deleted]

Re: How we spent $30k in Firebase in less than 72 hours

#106

Post by fefb » Thu, Aug 02, 2018, 5:10 AM UTC

superfrank wrote: post #101

t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

> This is why infinitely scaling pay-as-you-go cloud services terrify me. It's been a little over a year since I used Firebase in production, so maybe this has changed, but the funny thing is Firebase DB doesn't infinitely scale, despite them advertising that it does. The Firebase DB caps out at 100k active connections and according to them (at the time) it's a technological limit on their part, so they cannot go hig…

The link posted is talking about the Firestore and not the Real Time. Firestore is in beta state that has a beta limit of 100k sim. connections, probably off beta it will be able to handle more connections. The Real Time has 100K limit and is a centralized db.

Re: How we spent $30k in Firebase in less than 72 hours

#107

Post by flukus » Thu, Aug 02, 2018, 5:15 AM UTC

Aloha wrote: post #67

ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.

I think "try-anything" would be a better characterization in this case. They had smoke pouring out of the engine and tried changing the tire.

Re: How we spent $30k in Firebase in less than 72 hours

#108

Post by lifthrasiir » Thu, Aug 02, 2018, 5:16 AM UTC

I increasingly feel that modern pay-as-you-go services are so opaque to consumers that it takes the individual employee's empathy (highly subjective) or the publicity like this (highly subjective and you have to be lucky as well) to fix any significant problem. Every time a post like this with a "happy" ending crawls into the HN front page, there would be hundreds or maybe thousands of "unhappy" endings out there.

Re: How we spent $30k in Firebase in less than 72 hours

#109

Post by coding123 » Thu, Aug 02, 2018, 5:19 AM UTC

thatjsguy wrote: post #63

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

Are you... in the JS ecosystem yourself?

I used to be in the Java ecosystem, the C# ecosystem, the PHP ecosystem... and I could have made the statement "the most professional cluelessness I’ve ever encountered was in the X ecosystem."

I think it's just an industry thing.

Re: How we spent $30k in Firebase in less than 72 hours

#110

Post by rspeer » Thu, Aug 02, 2018, 5:20 AM UTC

paulddraper wrote: post #84

thatjsguy wrote: post #63

Earlier quoted context omitted.

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

It has to do with the barrier to entry. PHP, Ruby on Rails, and jQuery are other technologies that had low barrier to entry received the attention of the "unwashed masses". This being 2018, JS has very low barrier to adoption (Have a web browser? You have a JS runtime.) and nature runs its course.

Oh come on. Tech does not need barriers to entry. It just needs debugging and improving one's code to also have a low barrier to entry.