How we spent $30k in Firebase in less than 72 hours • Hacker News Board

Re: How we spent $30k in Firebase in less than 72 hours

#201

Post by gruez » Thu, Aug 02, 2018, 1:00 PM UTC

flukus wrote: post #136

ericpauley wrote: post #102

I've been noticing a steady rise of posts from hackernoon by amateur developers who think they'll be the next great tech blogger. I'm not saying I could do any better, but why are these posts suddenly getting so much attention?

> but why are these posts suddenly getting so much attention? The same reason we slow down for car crashes, morbid curiosity. I don't think there is anything "sudden" about it though, we even have sites like thedailywtf dedicated to this level of idiocy.

thedailywtf: making fun of other dvelopers' incompetence

hackernoon: author patting themselves on the back while everybody else is laughing at their incompetence

Re: How we spent $30k in Firebase in less than 72 hours

#202

Post by gcbirzan » Thu, Aug 02, 2018, 1:02 PM UTC

jsmthrowaway wrote: post #28

mirimir wrote: post #5

But wait, the image shows ~$600 million. So $30K is small. Maybe that ~$600 million isn't in USD?

No, it doesn’t. In many other locales “mil” means thousand, unlike slang for million. This is why in finance, $5mm means five million dollars. Five mil mil. Five thousand thousand. Five million. The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a…

> This is why in finance, $5mm means five million dollars

That's not true. It's from Latin, mille.

That aside, I find it a terrible way of writing things. But then again, Americans hate SI, so I guess have fun. :-)

Re: How we spent $30k in Firebase in less than 72 hours

#203

Post by jliptzin » Thu, Aug 02, 2018, 1:02 PM UTC

t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

I will also stay far away from these services. As a child post points out, if you do set a cap, you run the risk of cutting off a legitimate spike in high quality traffic. How do you tell the system to shut off due to an error and stay on during a spike in normal traffic? You can’t, unless you have someone watching it at all hours.

That’s why I prefer to just rent whole machines on AWS. If I accidentally have some code stuck in an infinite loop making some O(1) call, don’t charge me $10,000 for that when it costs you nothing. If it’s actually consuming electricity and significant resources, I’ll know quickly because my service will go down, as it should, not scale infinitely until my company is bankrupt.

Re: How we spent $30k in Firebase in less than 72 hours

#204

Post by douglaswlance » Thu, Aug 02, 2018, 1:03 PM UTC

r3vrse wrote: post #76

I spend a fair amount of time on HN. Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read. It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window. Why learn to thin…

This is the result of top-down learning. You learn the very latest tech and work your way down to the metal as required.

Bottom-up learning starts at the metal, at the very fundamentals of computation, and builds upward.

Re: How we spent $30k in Firebase in less than 72 hours

#205

Post by degenerate » Thu, Aug 02, 2018, 1:08 PM UTC

itcmcgrath wrote: post #25

shaunpersad wrote: post #6

I can't say enough good things about Firebase and GCP in general, but I'm always cautious when using Firestore in particular. I usually avoid unbounded queries altogether, and treat it primarily as a key/value store to get by id. When I do use queries, it's always in places where the results have a well-defined limit (usually limit = 1), e.g. finding the most recent X or the highest X. With the above two, you get all…

We've also been rolling out updates to rules to enable you to enforce these types of things. There is performance implications to limit queries for the real-time update system at scale, but for most use cases this shouldn't be a problem. Definitely more we can improve here for control, and we're open to feedback.

My open feedback is to get these rules in place as soon as possible. This story/blog makes me realize that GCP/Firebase does not currently play nice to dumb mistakes and 24h away from the dashboard can be disastrous.

Multiple rules/filters need to exist to trigger SMS/Email alerts, or a pre-defined action, upon certain conditions.

Re: How we spent $30k in Firebase in less than 72 hours

#206

Post by macspoofing » Thu, Aug 02, 2018, 1:12 PM UTC

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

>When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use.

Yeah. That jumped out at me as well. They spent an inordinate amount of effort to solve a non-problem. It's great to stay on evergreen with versions, but probably not a good thing to do so while you're desperately trying to debug a problem.

I suspect this was a hopeful but lazy attempt - in the spirit of "Maybe if we just do this, it will somehow fix the underlying problem". It's a lazy approach to solving problems. Debugging performance bottlenecks is hard and devs generally hate doing it. Upgrading version dependencies is a known factor and developers are comfortable with that.

Re: How we spent $30k in Firebase in less than 72 hours

#207

Post by gcbirzan » Thu, Aug 02, 2018, 1:12 PM UTC

Xorlev wrote: post #95

diziet wrote: post #79

Earlier quoted context omitted.

It's only 460k QPS, with 16k documents everything would be cached really well. Or a single instance of redis can serve that load of reads fairly easily.

Good luck with that. A single Redis would not be able to serve that workload. Maybe a single machine but you're really pushing the limits there just with concurrent TCP connections. At that qps Redis has 2 microseconds per request. I agree it caches well but your proposed architecture is definitely not production quality.

There is a difference between queries and connections. With pipelining, a redis machine can easily serve that. If they stored things in a better structure, like a list, they'd easily be able to get that.

On my machine 5 concurrent requests, at 100 items, it can do ~6 500 000 items per second, at 300, 11 000 000 per second, and it kind of caps out at that. Even with 1 concurrent connection, at 600 items, you get 6M per second.

Re: How we spent $30k in Firebase in less than 72 hours

#208

Post by jliptzin » Thu, Aug 02, 2018, 1:15 PM UTC

stickfigure wrote: post #127

alasdair_ wrote: post #14

Earlier quoted context omitted.

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) a…

This is a pretty common train of thought, but it's not really a question of deciding between "my bill will explode!" and "my app will go down!" You just really have to put some serious thought into what your daily limits should be, and add some reasonable alerting to detect surges. The tools are there and they're not terribly hard to use. It just tends to be an afterthought for most developers because this doesn't lo…

No, if your app is failing in a way that a page view now costs $1.00 instead of $0.0001 for example then the application should go down.

Re: How we spent $30k in Firebase in less than 72 hours

#209

Post by zebraflask » Thu, Aug 02, 2018, 1:16 PM UTC

yashap wrote: post #90

ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

2-7 are sort of understandable for a quick, hacky startup just trying to ship something fast, with minimal experience. But 1 is the really crazy one. Load times spiking to 30 seconds once they start getting significant traffic, and instead of doing a solid investigation, including instrumenting their FE and-or server so they can see where the slowness is (maybe Firebase even includes decent observability by default?)…

That raised an eyebrow for me, too. I recently moved a project from Angular 1 to 6, so this is fresh in my mind. There aren't enough changes between 4 to 6 (or even 1 to 6) to cause mysterious 30 second lag times by themselves. And upgrading the front end's base framework under a time crunch is almost always a bad decision, regardless of which framework you're using.

In the companies I've worked for, these guys would be written up and likely put on a performance improvement plan, if not flatly fired.

Re: How we spent $30k in Firebase in less than 72 hours

#210

Post by macspoofing » Thu, Aug 02, 2018, 1:17 PM UTC

overcast wrote: post #200

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

There has been MANY times, upgrading outdated libraries or software versions resolved similar performance issues for me. Should you just immediately run update all? No. But acting like that's not a viable solution is silly.

>But acting like that's not a viable solution is silly.

Usually you want to understand the problem before solving it. In this case, they wasted a bunch of time doing a bunch of things (upgrading all the dependencies, and refactoring the app) in the hope that something (ANYTHING) they're doing hopefully fixes a problem they don't understand. Smart move?