Why I’m dumping Firebase for Web

Post by mbell » Wed, Aug 09, 2017, 2:43 PM UTC

asadlionpk wrote: post #65

pjc50 wrote: post #59

Earlier quoted context omitted.

Hmm. Congratulations, you've nerd-sniped me this morning! This sounds like a classic use case for a streaming parser. The data is a mile wide and an inch deep, so at any point the memory requirements should not be too high. What do you want to do when you've parsed it? Insert it into a real database? Iterate over it? Would simply turning it into a list of user IDs one per line suffice?

Hello. The JSON dump looks something like this: { users: { "userid1": {...}, ... }, ... } I have tried jq stream parser to split the big dump into files like: - users.json - chatrooms.json - ... So I can then work on individual nodes. But jq fails silently after 12-24 hours of processing. I am still researching this in free time. If I can just get the keys (like "userid1") I can do the rest from firebase itself.

  cat input.json | jq -c --stream '. as $in | select(length == 2 and $in[0][0] == "users") | {}|setpath($in[0][1:]; $in[1])' > users.jsonlines

  {"userid1":{"name":"user1"}}
  {"userid2":{"name":"user2"}}

Post by skrebbel » Wed, Aug 09, 2017, 2:55 PM UTC

TeMPOraL wrote: post #86

skrebbel wrote: post #36

Earlier quoted context omitted.

> The problem I have is that someone took this cool project and basically lied to the public about its capabilities. Oh come on, it was a startup. The same people who made the tool told the lies (if that's what they were). Now under Google, this is still the case - the original founders now lead the Firebase department if I'm not mistaken. I'm pretty sure they have influence over the marketing messages.

> Oh come on, it was a startup. And then people are surprised when I advise them to avoid startup products. It's the classic tragedy of the commons; lying your way into an acquihire is a win for the successful founders, and a loss for the everyone else in the ecosystem, as more and more people grow sick of being played that way.

Post by asadlionpk » Wed, Aug 09, 2017, 3:10 PM UTC

mbell wrote: post #101

asadlionpk wrote: post #65

Earlier quoted context omitted.

Hello. The JSON dump looks something like this: { users: { "userid1": {...}, ... }, ... } I have tried jq stream parser to split the big dump into files like: - users.json - chatrooms.json - ... So I can then work on individual nodes. But jq fails silently after 12-24 hours of processing. I am still researching this in free time. If I can just get the keys (like "userid1") I can do the rest from firebase itself.

I would think something like this would work: cat input.json | jq -c --stream '. as $in | select(length == 2 and $in[0][0] == "users") | {}|setpath($in[0][1:]; $in[1])' > users.jsonlines Output would me a file that looks like this: {"userid1":{"name":"user1"}} {"userid2":{"name":"user2"}}

Post by Mc_Big_G » Wed, Aug 09, 2017, 3:14 PM UTC

Post by intrasight » Wed, Aug 09, 2017, 3:45 PM UTC