Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Firebase outages and misleading status reporting

medium.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Firebase outages and misleading status reporting

#21

Post by latchkey »

mattbillenstein wrote: post #2

AppEngine had the same problems - seemingly every week some component of the service would be down for some non-negligible amount of time (laughably it was often search -- we're talking about Google here). I've generally found AWS more reliable than GCP - even when GCP isn't having downtime, you'll occasionally get 503's from their APIs, so you need to wrap all your calls to them in retries. AWS has had multiple inst…

AppEngine had some big failures early on, but I (and some friends) built a $$$$$$$$$$$$ company on AppEngine (and GCP) and couldn't have done it without it. The stability the last few years has been extremely good. Our base logic was that we trust Google to hire and train talented DevOps more than we can do it and it sure sucks carrying a pager.

Re: Firebase outages and misleading status reporting

#22

Post by dahart »

sampl wrote: post #15

Not really convinced firebase is “covering it up”. The official status page breaks down availability by-service with descriptions of each outage and updates with timestamps. https://status.firebase.google.com

> The official status page breaks down availability by-service

That’s part of the problem, actually. I’ve noticed for years that some Firebase service distruptions go unreported, and it was clear that reporting individual services was a way to avoid showing the end-to-end summary. It doesn’t matter that all of Firebase’s servers are up and running, if the end-to-end service they provide isn’t working.

Re: Firebase outages and misleading status reporting

#23

Post by anothergoogler »

wild_preference wrote: post #17
jondubois wrote: post #11

I think that PaaS and BaaS where you don't have access to the back end is a dead end. It's going to go the way of Windows Server. Open source solutions will always win in the end when it comes to developers.

People also balked during the transition away from FTP. SSHing into servers is precisely the thing you want to get away from whether it's to change code or to hotfix your nginx.conf or to do a quick apt-get install. Doesn't mean that we don't need SSH ever, but 99% of the time it's something we use because we're too lazy to setup automation. I reckon you're using open-source here to mean self-hosted, but that doesn't…

"FTP -> SSH -> proprietary console" does not look like evolution over a gradient of control to me. I don't understand why you're comparing FTP to SSH when SSH is lower level than FTP. FTP "throw it on the server and let mod_php deal with it" deployments were decidedly higher level than SSH-based ones. FTP deployments were often coupled with GUI-based steps, for instance database migrations run from Drupal web app.

Re: Firebase outages and misleading status reporting

#24

Post by joeblau »

dotmanish wrote: post #18

I stopped using the realtime database once firestore was released in beta. So haven't experienced the downtime you have demonstrated in the status graphs, but Firebase's SLA [1] for realtime database apparently guarantees service credit for monthly uptime less than 99.95%. To corroborate your observations, check if you received this credit: Less than 99.95% but equal to or greater than 99.0%: 10% credit Less than 99.…

Is Firestore a more reliable version of Firebase's real-time database?

Re: Firebase outages and misleading status reporting

#25

Post by jondubois »

cortesoft wrote: post #20
jondubois wrote: post #11

I think that PaaS and BaaS where you don't have access to the back end is a dead end. It's going to go the way of Windows Server. Open source solutions will always win in the end when it comes to developers.

You are always going to have to rely on other service providers for critical things - networking, power, etc. I don’t think there is going to be some massive move for every business to be in control of every aspect of their supply chain. It simply isn’t feasible.

It's different. If a business has the option to do something themselves and doing so would cost them less in the long run and give them more flexibility, then doing it themselves is a competitive advantage.

If having solar panels becomes consistently cheaper than buying electricity from the grid (per megawatt), then individuals and businesses will all switch to solar panels... Especially if the business uses a lot of electricity.

The main reason that PaaS solution are popular now is because of advertising and hype. It's a bubble.

Re: Firebase outages and misleading status reporting

#26

Post by ankit219 »

yeah, we have suffered too. Initially we were using firebase Real Time DB for authentication as well as delivering messages. Messages suffered outages every now and then (and we suffered more cos our backend is in Python Django and Pyrebase comes with its own set of issues on top of Firebase). When we found out messages arent being delivered, we switched to pusher as a backup first and then to websocket. Now we use Firebase only for authentication (via real time database) and Notification sending, and still have a backend/app trigger every time there is an error on firebase.

I have always wondered what a reliable backup to the realtime db could be. Havent found much till date.

Re: Firebase outages and misleading status reporting

#27

Post by dotmanish »

joeblau wrote: post #24
dotmanish wrote: post #18

I stopped using the realtime database once firestore was released in beta. So haven't experienced the downtime you have demonstrated in the status graphs, but Firebase's SLA [1] for realtime database apparently guarantees service credit for monthly uptime less than 99.95%. To corroborate your observations, check if you received this credit: Less than 99.95% but equal to or greater than 99.0%: 10% credit Less than 99.…

Is Firestore a more reliable version of Firebase's real-time database?

It's a different database altogether - document-oriented at that.

https://firebase.google.com/docs/firestore/rtdb-vs-firestore

Re: Firebase outages and misleading status reporting

#28

Post by Guidii »

dahart wrote: post #22
sampl wrote: post #15

Not really convinced firebase is “covering it up”. The official status page breaks down availability by-service with descriptions of each outage and updates with timestamps. https://status.firebase.google.com

> The official status page breaks down availability by-service That’s part of the problem, actually. I’ve noticed for years that some Firebase service distruptions go unreported, and it was clear that reporting individual services was a way to avoid showing the end-to-end summary. It doesn’t matter that all of Firebase’s servers are up and running, if the end-to-end service they provide isn’t working.

Firebase offers a variety of individual services, and most apps pick up only the services they need. So reporting service-by-service makes more sense.

Re: Firebase outages and misleading status reporting

#29

Post by dahart »

Guidii wrote: post #28
dahart wrote: post #22

Earlier quoted context omitted.

> The official status page breaks down availability by-service That’s part of the problem, actually. I’ve noticed for years that some Firebase service distruptions go unreported, and it was clear that reporting individual services was a way to avoid showing the end-to-end summary. It doesn’t matter that all of Firebase’s servers are up and running, if the end-to-end service they provide isn’t working.

Firebase offers a variety of individual services, and most apps pick up only the services they need. So reporting service-by-service makes more sense.

That’s true, and beside the point. The problem isn’t reporting individual services, the problem is giving the impression that uptime for individual services equals uptime for Firebase as a whole.

Re: Firebase outages and misleading status reporting

#30

Post by burtonator »

I've been thinking about implementing Firebase as part of Polar: https://getpolarized.io/

The idea is that you update your documents (PDF, HTML, etc) into Polar, tag them, and then we sync them to the cloud. Then when you go to another machine like work or home your documents are always synchronized.

At first I fell in love with Firebase and was very very excited to start implementing it.

They've spent a ton of time working on the initial implementation experience.

Their Firebase Auth support was amazingly simple to setup. Same with Firebase hosting. It's top notch. You can be up and running with a CDN hosting with SSL in like 2 minutes and the firebase tools are exceptional.

Cloud Firestore seems really interesting and easy to setup. It's basically designed for 'apps'. IE user-facing apps and works pretty well if all the data is private to the user.

I do struggle with these issues of reliability though. At Datastreamer (http://www.datastreamer.io/) we use Hetzner and have about a half petabyte stored there.

It's a blog content search engine which we license to other startups so high availability is critical.

Their infra is amazingly reliable. Very very happy here.

The problem of course is that you then have to manage your own software stack which of course requires extra effort on your part.

Post reply on HN