361–370 of 416 posts
Earlier quoted context omitted.
Ah I see, we are going with “well technically it stores something therefore it is database joke”. Now I’m fully onboard :) Back when I worked in LA my CTO used to joke that most places use Microsoft Outlook as a database and Excel as BI tool.
You laugh, but I bet Excel produces orders of magnitude more real "business intelligence" than all other "BI" tools combined.
Earlier quoted context omitted.
“It doesn’t keep statistics” is a weird way to say “I expect full table scans to always be fast”. Create a functional index.
I’m not sure if I can even understand what you are talking about or why you said that. If you have pid,{name:’val’, others...}. And an index of name with a million John and one Jane. Good luck getting fast results.
Fast results: https://dbfiddle.uk/?rdbms=postgres_10&fiddle=dd9370966b1528....
Postgres does actually keep statistics on json columns, but if you've got a functional index on the table and the query uses it then it doesn't matter if there is one "jane" and a million "johns". You're looking up a key in a btree index.
MongoDB is horrible, I get it. What do I use in this situation: 1) I need to store 100,000,000+ json files in a database 2) query the data in these json files 3) json files come from thousands upon thousands of different sources, each with their own drastically different "schema" 4) constantly adding more json files from constantly new sources 5) no time to figure out the schema prior to adding into the database 6) d…
Depends on what your queries look like, I guess.
Earlier quoted context omitted.
"That's nice" is a way of explicitly stating that you don't care to any degree about whatever the other person just said.
What if it really is actually nice?
Earlier quoted context omitted.
Can you elaborate? I've seen benchmarks and from their website what I understood is that it can handle really massive reads and writes, tens (maybe hundreds) thousands of ops per second, but personally never tested to this extent.
We're using it in Quassel, and as soon as you go over ~3-4 parallel write/read threads, it starts locking up completely, sometimes taking 30 seconds for simple queries that should really take milliseconds. The big issue is that sqlite does full db locking for any operation, so during any write you can't just easily read at all. This can be fixed with WAL mode, but WAL mode is broken in uts early versions, and new ver…
MongoDB is horrible, I get it. What do I use in this situation: 1) I need to store 100,000,000+ json files in a database 2) query the data in these json files 3) json files come from thousands upon thousands of different sources, each with their own drastically different "schema" 4) constantly adding more json files from constantly new sources 5) no time to figure out the schema prior to adding into the database 6) d…
Elasticsearch? http://smnh.me/indexing-and-searching-arbitrary-json-data-us... Depends on what your queries look like, I guess.
Ironically once because mongo was such a pain to work with I dumped the data from it into ES to get the better API, usability and Kibana.
MongoDB is horrible, I get it. What do I use in this situation: 1) I need to store 100,000,000+ json files in a database 2) query the data in these json files 3) json files come from thousands upon thousands of different sources, each with their own drastically different "schema" 4) constantly adding more json files from constantly new sources 5) no time to figure out the schema prior to adding into the database 6) d…
Earlier quoted context omitted.
I don't think anyone posted about their particular system, but it's not unknown now. If you google "filesystem as a database" there are some relevant hits. One super simple and probably not ideal, but at least balanced version uses a hash of some primary key like customer row id as the file index, then partitions the items into directories with all permutations at each level (or only populated ones) based on successi…
One important note: make sure you carefully consider using atomic renames and such for manipulating the files! Overwrite in place is a great way to end up with a corrupted item if something goes desperately wrong and you're not protected by COW or data journaling.