Querying Postgres Tables Directly from DuckDB
1–10 of 41 posts
Re: Querying Postgres Tables Directly from DuckDB
#2This might end up being the best way to etl postgres tables to parquet. From everything else that I tried, doing a copy to CSV and then converting to parquet was the fastest but can be a pain when dealing with type conversions.
Re: Querying Postgres Tables Directly from DuckDB
#3Does the inverse of this also exist?
Re: Querying Postgres Tables Directly from DuckDB
#4Does the inverse of this also exist?
You can use the DuckDB FDW (linked in the blog post) for this!
https://github.com/alitrack/duckdb_fdw
Re: Querying Postgres Tables Directly from DuckDB
#5Does the inverse of this also exist?
Unofficially maybe? There's a 3rd party duckdb_fdw plugin for postgres that allows querying DuckDB from Postgres: https://github.com/alitrack/duckdb_fdw
Re: Querying Postgres Tables Directly from DuckDB
#6I wasn't aware of DuckDB but now that I read about I wonder should I replace my SQLite drivers with it, since it seems more versatile?
Re: Querying Postgres Tables Directly from DuckDB
#7This might end up being the best way to etl postgres tables to parquet. From everything else that I tried, doing a copy to CSV and then converting to parquet was the fastest but can be a pain when dealing with type conversions.
I didn't know DuckDB could read parquet natively[0]. Thanks for mentioning this. I recently had the displeasure of trying to deal with parquet files in nodejs and it was a disaster. I might try going through DuckDB instead.
Re: Querying Postgres Tables Directly from DuckDB
#8I wasn't aware of DuckDB but now that I read about I wonder should I replace my SQLite drivers with it, since it seems more versatile?
It’s pretty cool, but I wouldn’t switch over unless you know you need a column store approach.
Re: Querying Postgres Tables Directly from DuckDB
#9I wasn't aware of DuckDB but now that I read about I wonder should I replace my SQLite drivers with it, since it seems more versatile?
It’s pretty cool, but I wouldn’t switch over unless you know you need a column store approach.
I would say, maybe not switch over if you need something that has been rock-solid for years and years.
If you want a shit-ton of built-in features DuckDB is pretty damn fast even with non-column store optimized queries. Surprisingly so! I have done side-by-side tests and you really have to have a lot of rows for it to start to make a difference. And their built-in functions are awesome. And they (company/group) are awesome and very responsive!
Having said that DuckDB is still young enough to make me nervous putting in critical application stacks and I feel about Sqlite the same way I feel about PostgreSQL ... it is solid as hell and will just work.
I will say that I have not had any stability issues in any of my personal projects that I use DuckDB in, it is just that I know for a fact that Sqlite is solid.
Re: Querying Postgres Tables Directly from DuckDB
#10DuckDB is awesome! I find it the easiest way to ingest data from various sources then query it into a form I can do analytics on.
The datasets I work on are a bit too big for pandas, but spark is way overkill for them. DuckDB lets me efficiently work on them using only a single computer.