41–50 of 104 posts
Anyone tried using the Go driver? It has a few open bugs: https://github.com/marcboeker/go-duckdb/issues And a third-party effort, only: https://duckdb.org/docs/api/overview
Isn’t SQLite adding features for analytic queries that should take the wind out of the sails of duckdb?
The SQL dialect is so much lacking that it seems intentional. They are meant to be a transactional database, not an analytics one.
Sqlite also takes pride on stability (deployed on a billion android devices). Adding 100+ analytics capabilities e.g. functions is not gonna be easy in terms of maintaining stability.
I want to be wrong though because my paid app (superintendent.app) uses Sqlite. Not supporting analytics well is the number one complaint.
Earlier quoted context omitted.
It’s a drop in alternative to SQLite that’s column-oriented/OLAP. I’ve been profiling entire projects in production switching between SQLite and duckdb (no clear conclusions yet)
I suppose that leads to a broader question: when should you use an in-memory database, and when should you use a data frame library? The distinction between the two seems to be getting blurry (which maybe is a good thing).
Earlier quoted context omitted.
I see. Would it be fair to say you treat it almost like Pandas, except that it has a lower memory footprint since data is written to disk instead of memory. IE you use it for on the fly analysis of large frames of data, not like more traditional database/datawarehouse?
I'm not sure they're saying that. BTW, your questions are exactly those that I've been ask over the last few months, but also with a lot of focus over the last few days. Still learning as much as I can so the following might not be true. For what it's worth, there's a difference between using duckdb to query a set of files vs loading a bunch of files in to a table. But once the data has been loaded into a table it ca…
Earlier quoted context omitted.
I'm not sure they're saying that. BTW, your questions are exactly those that I've been ask over the last few months, but also with a lot of focus over the last few days. Still learning as much as I can so the following might not be true. For what it's worth, there's a difference between using duckdb to query a set of files vs loading a bunch of files in to a table. But once the data has been loaded into a table it ca…
Ah, almost like a little bit of a lower level trino, where DuckDB is able to push out queries to different data storage endpoints?
https://www.arecadata.com/getting-started-with-iceberg-using...
You would still need to interact with some kind of catalogue to understand which .db files you need to fetch.
And honestly I don't really know or understand the performance implications of the attach command.
I'm excited to see if the duckdb team will be able to integrate with external tables directly one day. (not that data files would be .db files)
Imagine this:
1) you have an external managed external table (iceberg, delta, etc... managed by Glue, databricks, etc)
2) register this table in duckdb
CREATE OR REPLACE EXTERNAL TABLE my_table ... TYPE = 'ICEBERG' CATALOG = 's3://...' CREDENTIALS = '...' etc
DuckDB is terrific. I'm bullish on its potential for simplifying many big data pipelines. Particularly, it's plausible that DuckDB + Parquet could be used on a large SMP machine (32+ cores and 128GB+ memory) to deal with data munging for 100s of gigabytes to several terabytes, all from SQL, without dealing with Hadoop, Spark, Ray, etc. I have successfully used DuckDB like above for preparing an ML dataset from about…
There's a nice intro to DuckDB for julia developers by Bogumił Kamiński, the creator of DataFrames.jl here: https://juliazoid.com/welcome-to-duckdb-3c4e75f50b97 . Interesting since in some ways, as he points out, it's in direct competition with DataFrames for use cases, but he gives it a very positive treatment and shows how they can work together using advantages of standard SQL along with processing power of DataFr…
When someone gives fair opinions on something that directly competes were their own work, you should take their opinion very seriously. It’s an excellent quality in a person, and shows they’re more focused on the problem than their ego.
The approach was never as fast as data.table but did approach the speed of dplyr for more complex queries.
Life had other things in store for me and I haven’t touched this library for a while now.
At the time there was no Julia connector for duckdb, but now that there is, I’d like to try this approach in that language.
I see no MDX query support. How is this an OLAP database? Or do I misunderstand what it is?
OLAP cube is microsoft's take on OLAP using 90s technologies for tech stacks from 1990s (Windows Server + SQL Server + SSAS).
DuckDB is a modern take on OLAP
I've written a lot of OLAP queries (wrote a materialization layer for MonetDb and Postgres years ago). I find Pandas so much easier to work with for semi complicated work.