I'm impressed to all heck how fast this is, and how intuitive!
I tried it against a large-ish (hundreds of mb) log file and the query was simple to write while spitting out the result .. very quick.
Impressed!
DuckDB: Querying JSON files as if they were tables
61–70 of 81 posts
Re: DuckDB: Querying JSON files as if they were tables
#62Re: DuckDB: Querying JSON files as if they were tables
#63How does this compare with CouchDB, which also stores JSON and allows you to construct cached map/reduce views?
edit: Looks like DuckDB lets you use SQL-style queries
Re: DuckDB: Querying JSON files as if they were tables
#64One of the major reasons we bet big on DuckDb at MotherDuck is due to the extendable nature of the code base, as demonstrated by the frequency of major improvements and additions.
Re: DuckDB: Querying JSON files as if they were tables
#65Really makes you wonder what's the difference between NoSQL and SQL databases if they can be rendered as each other.
Re: DuckDB: Querying JSON files as if they were tables
#66If you ever need to join two large dataframes, but are OOMing on the join, write them to disk as parquet files then use DuckDB to do the join. It's amazing what you can do on one machine thanks to DuckDB.
Re: DuckDB: Querying JSON files as if they were tables
#67If you ever need to join two large dataframes, but are OOMing on the join, write them to disk as parquet files then use DuckDB to do the join. It's amazing what you can do on one machine thanks to DuckDB.
This isn't unique to duckdb. Almost all databases allow for sorting and joins of large tables that don't fit into memory.
Re: DuckDB: Querying JSON files as if they were tables
#68Very nice! Does anyone know if we can query duckdb with a pandas dialect?
Hey, I maintain a tool called siuba that converts pandas methods to SQL, including for duckdb!
* duckdb example: https://siuba.org/guide/workflows-backends.html#duckdb
* supported methods: https://siuba.org/guide/ops-support-table.html
Re: DuckDB: Querying JSON files as if they were tables
#69Earlier quoted context omitted.
Tried to open DuckDB file with one table with 890K rows and about 30 columns. Process stalled, had to kill it (MacOS). DuckDB cli opens it in a snap.
:( Thanks for trying. The only thing I can think is perhaps a version conflict (requires 0.70) but I would have expected it to say, rather than freeze. I will try to recreate the problem. Thanks.
Yes, I think I have 0.60, good point. I'll update the version and try again later.
Re: DuckDB: Querying JSON files as if they were tables
#70Ah I was looking for exactly this the other day. I'm try to build a git based interface to our BI tool so that we can get config for our reports in source control instead of configuration in a db. Was looking for something to read json files which will house the config via SQL, i.e. a human readable db as an alternative to what the BI tool is using for it's config persistence. Will give DuckDB a go, thanks for postin…
Are you talking about metabase by any chance?