Firebase outages and misleading status reporting • Hacker News Board

Re: Firebase outages and misleading status reporting

#31

Post by dahart » Sat, Oct 13, 2018, 4:30 PM UTC

iamleppert wrote: post #7

Serves you right for using a “real-time database” (whatever that is). I’m sure your chat product feature could have been designed using a flat file as a datastore and a simple web socket server.

> Serves you right for using a “real-time database” (whatever that is).

I see this is flagged, but FWIW, you might want to actually learn something about what they mean by “realtime database” because it’s incredibly useful, and people using Firebase aren’t the only people who think so.

https://en.m.wikipedia.org/wiki/Real-time_database

Firebase is also easy to use and scales to large sites and complex applications, despite the complaints here about reliability, reporting and control, or lack of. A flat file and simple web socket server crumbles under loads that Firebase handles easily.

Re: Firebase outages and misleading status reporting

#32

Post by nslog » Sat, Oct 13, 2018, 4:40 PM UTC

ankit219 wrote: post #26

yeah, we have suffered too. Initially we were using firebase Real Time DB for authentication as well as delivering messages. Messages suffered outages every now and then (and we suffered more cos our backend is in Python Django and Pyrebase comes with its own set of issues on top of Firebase). When we found out messages arent being delivered, we switched to pusher as a backup first and then to websocket. Now we use F…

AWS AppSync and Amplify

Re: Firebase outages and misleading status reporting

#33

Post by xrd » Sat, Oct 13, 2018, 4:44 PM UTC

Have to say, having worked in a huge organization with multiple clients accessing services, I much prefer the firebase solution. You still have downtime in any polyglot solution and the problem is pretty clear here (it's firebase database, not one of dozens of legacy layers...). When you own the entire stack it is amazing how much of the organizational effort goes into obscuring who is responsible. And the stack is much more opaque.

It really is possible to design a system around firebase with a much smaller team. You give up control but control is a myth anyway. And, Firestore is actually designed to support offline mode, so wonder if they neglected to design for that feature which might help here.

The unfortunate reality is that we are in a moment where Firestore is beta and Firebase Database is not supported as it should be. Google should do a better job of helping people to migrate and explaining the roadmao. I imagine the writer of this article just doesn't have as much company clout to get that level of involvement from Google. This was probably an attempt to get that attention that other higher paying clients can get.

Re: Firebase outages and misleading status reporting

#34

Post by 013a » Sat, Oct 13, 2018, 4:53 PM UTC

sampl wrote: post #15

Not really convinced firebase is “covering it up”. The official status page breaks down availability by-service with descriptions of each outage and updates with timestamps. https://status.firebase.google.com

From the article: "When I tell our customers something is wrong outside of our control"

I think this is both the issue with the article, and the issue with Firebase (ironically).

First of all, its an issue with Firebase. All software will break. This is inevitable. Its just a matter of time. Well engineered software/infrastructure gives you, the consumer, tools to mitigate this so your consumers never see it. If we look at amazon, they expose AZs and Regions; well architected applications use these failure domains to accept that an AZ, and possible even a region, will fail. So you can do fallover.

Firebase really doesn't expose these primitives, in an effort to be simple and easy to use. Maybe they're doing something in the backend to use them, but the proof is in the pudding; if their stability is bad, it means they're not doing a good enough job at abstracting away these unavoidable failure domain principals.

Which brings us to the second problem: Its Always Your Fault. Stop trying to pass blame to Firebase. Your customers, seriously, full stop, unequivocally, no exceptions, do not care that Firebase caused you to go down. They care that you went down. You don't get to say "its not our fault!"

Because Its Always Your Fault. Its your fault that you chose Firebase. Its your fault that you chose a service which doesn't expose core failure domain primitives that you can engineer to support. Its your fault for not getting off Firebase when you recognize these core issues with the platform.

Firebase's status page is for you, the engineer, to understand and diagnose issues. Its for you to interpret and surface on your own status page. Its not for you to link to your customers and say "see that red dot? that's why we went down."

And by the way, Yes: Even perfectly architected applications on AWS/Gcloud/Whatever, falling over AZs and Regions, can go down due to things outside of your control. AWS ain't perfect. Remember: All software breaks. But when you word that to your customer, You Always Take The Blame. Period. This is what "its always your fault" means; its not about saying that there are ways to write an application that never breaks. Its about accepting that when (not if) it does break, your customers will blame you, so you need to accept that blame wholly.

Re: Firebase outages and misleading status reporting

#35

Post by novaleaf » Sat, Oct 13, 2018, 5:14 PM UTC

I think that now, Firebase is build on Google Cloud Datastore. I have used Datastore in production since 2015, and have had no outages, but if I had to do it again I think I'd go normal RDB, just because query support is extremely limited (no full text search) and "schema change == data rebuild" issues.

Re: Firebase outages and misleading status reporting

#36

Post by dfee » Sat, Oct 13, 2018, 5:19 PM UTC

novaleaf wrote: post #35

I think that now, Firebase is build on Google Cloud Datastore. I have used Datastore in production since 2015, and have had no outages, but if I had to do it again I think I'd go normal RDB, just because query support is extremely limited (no full text search) and "schema change == data rebuild" issues.

Do you mean that you’d use something like a managed Postgres AND build and run a backend service that interfaces a web client to that database?

Re: Firebase outages and misleading status reporting

#37

Post by romed » Sat, Oct 13, 2018, 5:28 PM UTC

Firebase RTDB is basically the legacy product that barely works. Firestore is the post-acquisition product built on Google tech. It’s a rotten situation. I noticed the outage mentioned in this post because it took down Ford GoBike (and Citibike and all the other Motivate/Lyft bike share systems).

Re: Firebase outages and misleading status reporting

#38

Post by kevan » Sat, Oct 13, 2018, 5:38 PM UTC

mattbillenstein wrote: post #2

AppEngine had the same problems - seemingly every week some component of the service would be down for some non-negligible amount of time (laughably it was often search -- we're talking about Google here). I've generally found AWS more reliable than GCP - even when GCP isn't having downtime, you'll occasionally get 503's from their APIs, so you need to wrap all your calls to them in retries. AWS has had multiple inst…

>you'll occasionally get 503's from their APIs, so you need to wrap all your calls to them in retries.

No matter which cloud platform you're using you should do this[1]. I'm not familiar with the GCP SDK but I know the AWS SDK has it built in[2]. If you're not using the SDK then you have to build it yourself. There will always be a small percentage of transient errors due to the network, DNS, timeouts, hardware failure, etc.

[1] This is a blanket generalization, there are some situations where you shouldn't use the backoff/retry pattern even for retryable errors.

[2] https://docs.aws.amazon.com/general/latest/gr/api-retries.ht...

Re: Firebase outages and misleading status reporting

#39

Post by cortesoft » Sat, Oct 13, 2018, 5:39 PM UTC

jondubois wrote: post #25

cortesoft wrote: post #20

Earlier quoted context omitted.

You are always going to have to rely on other service providers for critical things - networking, power, etc. I don’t think there is going to be some massive move for every business to be in control of every aspect of their supply chain. It simply isn’t feasible.

It's different. If a business has the option to do something themselves and doing so would cost them less in the long run and give them more flexibility, then doing it themselves is a competitive advantage. If having solar panels becomes consistently cheaper than buying electricity from the grid (per megawatt), then individuals and businesses will all switch to solar panels... Especially if the business uses a lot of…

That is interesting, because the advice I always hear for businesses is to keep in house their core business and contract/outsource everything else. That 'bring everything in house' strategy only works for the biggest companies that have enough scale.

For almost everyone else, the cost of providing a profit margin to the contracting company is dwarfed by the savings you get from the economy of scale the contractor is able to provide.

Really, this is the microeconomics version of the ideas behind free trade. It is better to produce what you are best at and trade for the rest.

Re: Firebase outages and misleading status reporting

#40

Post by romed » Sat, Oct 13, 2018, 5:43 PM UTC

mattbillenstein wrote: post #2

AppEngine had the same problems - seemingly every week some component of the service would be down for some non-negligible amount of time (laughably it was often search -- we're talking about Google here). I've generally found AWS more reliable than GCP - even when GCP isn't having downtime, you'll occasionally get 503's from their APIs, so you need to wrap all your calls to them in retries. AWS has had multiple inst…

Retrying localizable errors (such as 503 Service Unavailable) should be universally practiced in any RPC scheme. Nobody can make a backend that's 100% reliable.