Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

121–130 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#121

Post by curun1r »

coding123 wrote: post #109
thatjsguy wrote: post #63

Earlier quoted context omitted.

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

Are you... in the JS ecosystem yourself? I used to be in the Java ecosystem, the C# ecosystem, the PHP ecosystem... and I could have made the statement "the most professional cluelessness I’ve ever encountered was in the X ecosystem." I think it's just an industry thing.

Having spent more than a decade working and hiring in the Java world, I'm inclined to agree.

Having spent much of the past year writing Rust and interacting with that community, I'm inclined to disagree.

My overall feeling is that it isn't only JavaScript, but it is JavaScript and languages that attract a higher caliber of developer don't suffer from this problem the way that languages that appeal to a wider demographic do.

Re: How we spent $30k in Firebase in less than 72 hours

#122

Post by emodendroket »

paulddraper wrote: post #84
thatjsguy wrote: post #63

Earlier quoted context omitted.

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

It has to do with the barrier to entry. PHP, Ruby on Rails, and jQuery are other technologies that had low barrier to entry received the attention of the "unwashed masses". This being 2018, JS has very low barrier to adoption (Have a web browser? You have a JS runtime.) and nature runs its course.

Something about jQuery does remind me of JavaScript

Re: How we spent $30k in Firebase in less than 72 hours

#123

Post by onion2k »

alasdair_ wrote: post #14
itcmcgrath wrote: post #11

Earlier quoted context omitted.

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) a…

Your app shouldn't just fail if an API is unavailable. You should code things to fail gracefully. In the case of a traffic spike replacing the front end with an email capture form and a message saying "things are really popular right now, how about we remind you about this tomorrow (with a discount code!)" works well.

This is hard for some things, but your startup failing because you didn't want to do it is much harder in the end.

Re: How we spent $30k in Firebase in less than 72 hours

#124

Post by dragonwriter »

noncoml wrote: post #58
ConcernedCoder wrote: post #54

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.

I wouldn’t call it architecture decision. It’s the equivalent of a bad SQL query.

It's more like a naive database design which doesn't consider the access pattern and therefore requires an inefficient query, which is definitely an architectural issue.

Re: How we spent $30k in Firebase in less than 72 hours

#125

Post by gitgud »

sitkack wrote: post #40
sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an…

Well a browser is basically a sandbox for a webapp.

To debug something complex, I would use chrome devtools, which can measure all kinds of metrics, and the function "Audit > LightHouse" automates the process and ranks a webapp in several key categories.

This case would appear to be related to network requests, so that issue should be fairly obvious in LightHouse.

Re: How we spent $30k in Firebase in less than 72 hours

#126

Post by onion2k »

r3vrse wrote: post #76

I spend a fair amount of time on HN. Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read. It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window. Why learn to thin…

Critical thinking is a learned skill. For most developers it comes with experience, and many developers in startups are often yet to learn it. Thinking well under pressure is even harder.

This is more a problem about startups using inexperienced developers than anything related to what they're building or which tech they're using.

Re: How we spent $30k in Firebase in less than 72 hours

#127

Post by stickfigure »

alasdair_ wrote: post #14
itcmcgrath wrote: post #11

Earlier quoted context omitted.

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) a…

This is a pretty common train of thought, but it's not really a question of deciding between "my bill will explode!" and "my app will go down!"

You just really have to put some serious thought into what your daily limits should be, and add some reasonable alerting to detect surges. The tools are there and they're not terribly hard to use. It just tends to be an afterthought for most developers because this doesn't look like a customer feature.

Without these sorts of automated scaled services, the traditional behavior is "your app goes down". This is a big improvement!

Re: How we spent $30k in Firebase in less than 72 hours

#128

Post by tkrupicka »

itcmcgrath wrote: post #11
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

Hey! This might not be the best place to contact you, but my team is currently in crisis from a nasty Android Firestore bug. When the user's device switches networks it loses the real-time event listeners and in some cases doesn't re-establish until the app is reinstalled. We submitted a bug report and source for an app with the issue but haven't heard anything in a week. We love the product otherwise, but I thought I'd bring it to your attention.

Re: How we spent $30k in Firebase in less than 72 hours

#129

Post by blub »

scrollaway wrote: post #85
thatjsguy wrote: post #63

Earlier quoted context omitted.

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

> the most professional cluelessness I’ve ever encountered was in the JS ecosystem. A testament to how low the barrier of entry has gotten. It's both a good and bad thing at the same time. It however leads to having to be ever more so vigilant about at least your first layer of dependencies in that ecosystem, if you do want to be professional. The higher the barrier of entry is to a language, the more likely it is th…

It's been a long time goal of the software dev community to lower barriers to entry through bootcamps and such, in an attempt to "democratize software".

And the result is unsurprisingly poorer quality software. So why is it a good thing?

Re: How we spent $30k in Firebase in less than 72 hours

#130

Post by bufferoverflow »

tgsovlerkhgsel wrote: post #80
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

How do pay-as-you-go cloud providers handle DoS attacks (the "smart" ones that simulate a lot of expensive legit-looking traffic, not simple volumetric ones)? That's the thing that terrifies me. If I'm using S3/Cloud Storage etc., I'm getting charged for each GB of outbound traffic, and I have to assume that the bandwidth available to serve my files is almost infinite.

You should rate-limit and cache the potentially expensive paths of your app. There's no real way to protect against a clever DDoS, someone will have to eat the costs.
Post reply on HN