Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I’m dumping Firebase for Web

lugassy.net

Post reply on HN
Print view View original

41–50 of 123 posts

Re: Why I’m dumping Firebase for Web

#41

Post by saganus »

asadlionpk wrote: post #37
superfrank wrote: post #24

I have had so many issues with Firebase, but the one that really pisses me off the most is that they used to advertise things like "scale worldwide to millions of users" and "Unlimited Connections" but then cap you at 100k concurrent users (10k initially and will gradually bump you up to 100k). When we got in contact with them about this they were nice enough to explain that we could create another database and shard…

That's not the only limit btw. If any node (say "/users/") crosses 'some' number of subnodes ("/users/a", "users/b") then you cannot do any queries on the node itself. Like I cannot get even get the IDs of the subnodes ("a","b",...). I also got similar advice to shard my users or something. So right now, we have crossed that limit and are unable to know how many users are on our system. Their server just fails and ta…

Wow.

Running a query takes down the db?

That sounds like a major problem. How can that happen?

Maybe returning incorrect or incomplete results due to sharding... But taking the db down? That's very... Unexpected.

Re: Why I’m dumping Firebase for Web

#42

Post by superfrank »

asadlionpk wrote: post #37
superfrank wrote: post #24

I have had so many issues with Firebase, but the one that really pisses me off the most is that they used to advertise things like "scale worldwide to millions of users" and "Unlimited Connections" but then cap you at 100k concurrent users (10k initially and will gradually bump you up to 100k). When we got in contact with them about this they were nice enough to explain that we could create another database and shard…

That's not the only limit btw. If any node (say "/users/") crosses 'some' number of subnodes ("/users/a", "users/b") then you cannot do any queries on the node itself. Like I cannot get even get the IDs of the subnodes ("a","b",...). I also got similar advice to shard my users or something. So right now, we have crossed that limit and are unable to know how many users are on our system. Their server just fails and ta…

> Their server just fails and takes down the DB for 10 or so mins if I do that query.

I don't know if you are aware, but their DB can't handle more than 1000 requests/sec, so if you are iterating through a list of nodes and requesting data for each one you can hit that limit (not a good practice, but sometimes you have to). Additionally, once you hit that limit, the DB slows down but keeps accepting requests meaning, if you keep hitting it, even at a slower rate you make the backlog worse. Seriously, be very very careful not to go over that limit, we found out the hard way.

Re: Why I’m dumping Firebase for Web

#43

Post by asadlionpk »

superfrank wrote: post #42
asadlionpk wrote: post #37

Earlier quoted context omitted.

That's not the only limit btw. If any node (say "/users/") crosses 'some' number of subnodes ("/users/a", "users/b") then you cannot do any queries on the node itself. Like I cannot get even get the IDs of the subnodes ("a","b",...). I also got similar advice to shard my users or something. So right now, we have crossed that limit and are unable to know how many users are on our system. Their server just fails and ta…

> Their server just fails and takes down the DB for 10 or so mins if I do that query. I don't know if you are aware, but their DB can't handle more than 1000 requests/sec, so if you are iterating through a list of nodes and requesting data for each one you can hit that limit (not a good practice, but sometimes you have to). Additionally, once you hit that limit, the DB slows down but keeps accepting requests meaning,…

Iterating would require me to first get atleast a 'shallow' list of keys for that node. But even that one REST query for shallow list of all keys crashes their server instance.

I am not even sure how I should get keys for all my user nodes anymore.

I tried doing it from an offline JSON backup they generate. But that one giant 100GB+ JSON is impossible to parse with any available tools.

Re: Why I’m dumping Firebase for Web

#45

Post by asadlionpk »

saganus wrote: post #41
asadlionpk wrote: post #37

Earlier quoted context omitted.

That's not the only limit btw. If any node (say "/users/") crosses 'some' number of subnodes ("/users/a", "users/b") then you cannot do any queries on the node itself. Like I cannot get even get the IDs of the subnodes ("a","b",...). I also got similar advice to shard my users or something. So right now, we have crossed that limit and are unable to know how many users are on our system. Their server just fails and ta…

Wow. Running a query takes down the db? That sounds like a major problem. How can that happen? Maybe returning incorrect or incomplete results due to sharding... But taking the db down? That's very... Unexpected.

Nope. A clear server crash with "Internal Server Error" and the DB being totally unavailable for 10-15 mins. Apparently it's 'normal'.

Re: Why I’m dumping Firebase for Web

#46

Post by stemuk »

wolframhempel wrote: post #28

(deepstream employee here) despite its flaws, Firebase has some great ideas, e.g. its permission language that strongly inspired us when building https://deepstreamhub.com/ . But there are some aspects, many mentioned by the OP that send us down a different route: - Serverless is a great goal, but only works for low complexity apps. We've designed deepstream explicitly not to take the server away, but as a layer that…

As far as scalability goes the open-source solution doesn't seem too promising. Beeing capped at 10.000 connections seems like a very low limit to me, especially considering that the blog post complained about 100.000 connections not beeing enough.

Re: Why I’m dumping Firebase for Web

#47

Post by calpaterson »

saganus wrote: post #41
asadlionpk wrote: post #37

Earlier quoted context omitted.

That's not the only limit btw. If any node (say "/users/") crosses 'some' number of subnodes ("/users/a", "users/b") then you cannot do any queries on the node itself. Like I cannot get even get the IDs of the subnodes ("a","b",...). I also got similar advice to shard my users or something. So right now, we have crossed that limit and are unable to know how many users are on our system. Their server just fails and ta…

Wow. Running a query takes down the db? That sounds like a major problem. How can that happen? Maybe returning incorrect or incomplete results due to sharding... But taking the db down? That's very... Unexpected.

Never used firebase but for what it's worth you can take down most databases with a bad enough query.

Re: Why I’m dumping Firebase for Web

#49

Post by stemuk »

superfrank wrote: post #24

I have had so many issues with Firebase, but the one that really pisses me off the most is that they used to advertise things like "scale worldwide to millions of users" and "Unlimited Connections" but then cap you at 100k concurrent users (10k initially and will gradually bump you up to 100k). When we got in contact with them about this they were nice enough to explain that we could create another database and shard…

It is definitely true that pagination could be much easier. The way it currently works however seems fine to me: You query a node with say the rate limit of the first 30 children. Then you store the key of the last result, and use it as the starting point for the next limited query. This way you can easily query a node with 1000+ children without overloading the client.

Re: Why I’m dumping Firebase for Web

#50

Post by skrebbel »

cultofmetatron wrote: post #4

http://phoenixframework.org/ actually has a built in websocket layer called channels. (its the inspiration for action cable and django channels) . Unlike deepstream, or socketcluster, it has a longpolling fallback as well as multi node clustering. Its a shame more people don't realize what a gold mine of a piece of technology it is.

Phoenix Channels are great, but they're totally not a replacement for Firebase. You need to do a lot of coding on top of Channels to get what Firebase offers you.

Not that that takes anything away from your argument, but given that this topic is about Firebase, readers could get the impression that it offers the same.

That said, I like Phoenix Channels too, and the long polling support is a must-have for us. The "very good architectural reasons" for not doing this, as quoted by @jondubois in a sister comment, are nice, but if that means that 10% of our audience can't use our product then that's a show stopper. I love that Phoenix takes these cases seriously.

That said, Phoenix Channels are a mess when it comes to documentation and terminology. It's built on only two terms, "socket" and "channel", both of which mean 2 to 3 different but related things.

(background: at https://talkjs.com we recently moved our realtime stuff from Firebase to Phoenix Channels)

Post reply on HN