I don't understand the purpose of this post. "I write a lot of X so I prefer using X over Y." Great.
DuckDB over Pandas/Polars
31–40 of 43 posts
Re: DuckDB over Pandas/Polars
#32I don't understand the purpose of this post. "I write a lot of X so I prefer using X over Y." Great.
It's an expression of a personal experience, preferences, and thoughts on a personal blog, thrown for others that might care about DuckDb and Pandas/Polars (and many did, as it got in the HN's first page).
They didn't write it to be some novel research, some canonical tutorial about the tech, or to teach/amuse each and every random reader.
Re: DuckDB over Pandas/Polars
#33Earlier quoted context omitted.
Where does your data reside, is it on an attached EBS volume, or in S3, or somewhere else? I had some spare time and tinkered with duckdb with a 70GB dataset, but just getting the 70GB on to the EC2 took hours. Would be pretty rocking if duckdb team could somehow set up a ~1TB sized demo that anyone can setup and try for themselves in, say, under an hour.
Local drives. DONT USE EBS! you’ll incur a huge IO charge. You have to choose instances with attached nvme storage which means one of the storage optimized instances. Reading the data off s3 will mean you will be slower than offerings like snowflake. Snowflake has optimized the crap out of doing analytics in s3, so you can’t beat it with something as simple as duckdb. Importantly you need the data in some distributed…
Hi – DuckDB Labs devrel here. It's great that you find DuckDB useful!
On the setup side, I agree that local (instance-attached) disks should be preferred but does EBS incur an IO fee? It incurs a significant latency for sure but it doesn't have a per-operation pricing:
> I/O is included in the price of the volumes, so you pay only for each GB of storage you provision.
Re: DuckDB over Pandas/Polars
#34Earlier quoted context omitted.
we use partitioned parquet files in s3. we use a csv in the bucket root to track the files. i’m sure there’s a better way but for now the 2tb of data are stored cheaply and we get fast reads by only reading the partitions we need to read.
I'm curious how much simpler to build, manage, and run vs cost it would be to simply running a database on a large vultr/DO instance and paying for 2tb of storage? I feel like you'd get away with the whole thing for around $500/mo depending on how much compute was needed?
well that's not the infrastructure we have. we are primarily an aws shop so we use the resources available to us in the context of our infrastructure decisions. it would be a hard sell to buy something outside of that ecosystem.
Re: DuckDB over Pandas/Polars
#35Earlier quoted context omitted.
I tried to spread large dataset into thousands of files on S3 and use StepFunctions Distributed Map to launch thousands of Lambda instances to process those files in parallel, using DuckDB (or other libs) in Lambda. The parallel loading and processing is way faster than doing this in a single big EC2 instance.
Lambda isn’t infinitely parallel. I thought it doesn’t do more than 100 parallel runners? I4i.metal has 96 cores and can be faster than that.
As per AWS said in https://aws.amazon.com/cn/blogs/aws/aws-lambda-functions-now...
> Each synchronously invoked Lambda function now scales by 1,000 concurrent executions every 10 seconds.
Re: DuckDB over Pandas/Polars
#36Why not both? https://ibis-project.org/
Re: DuckDB over Pandas/Polars
#37I think the competition for the future is between DuckDB and Polars. Will we stick with the DataFrame model, made feasible by Polars's lazy execution, or will we go with in-process SQL a la DuckDB? Personally I've been using DuckDB because I already know SQL (and DuckDB provides persistence if I need it) and don't want to learn a new DataFrame DSL but I'd love to hear other the experience of other people.
I've written a fair bit of PySpark code and Polars's syntax feels fairly similar, but it also offers a limited SQL dialect.
Although only experimental and probably off topic to the discussion, it's worth mentioning DuckDB also provides a Spark API implementation.
https://duckdb.org/docs/api/python/spark_api
And while on the subject of syntax, duckdb also has function chaining
https://duckdb.org/docs/sql/functions/overview.html#function...
Re: DuckDB over Pandas/Polars
#38The test case of a simple aggregation is a good example of an important data science skill knowing when and here to use a given tool, and that there is no one right answer for all cases. Although it's worth noting that DuckDB and polars are comparable performance-wise for aggregation (DuckDB slightly faster: https://duckdblabs.github.io/db-benchmark/ ). For my cases with polars and function piping, certain aspects of…
The real winner is going to be a framework that, during dev, transparently materializes CTEs to temporary tables so you can iterate on them like you’re saying, while continuing to harness SQL for the end product.
Perhaps not exactly what you're talking about, but maybe? (unsure bc the with statements are sometimes called "temp tables")
https://duckdb.org/docs/sql/query_syntax/with#cte-materializ...
Obviously, the materialization is gone after the query has ended, but still a very powerful and useful directive to add to some queries.
There are also a few DuckDB extensions for pipeline SQL languages.
https://duckdb.org/community_extensions/extensions/prql.html
https://duckdb.org/community_extensions/extensions/psql.html
And of course dbt-duckdb https://github.com/duckdb/dbt-duckdb
Re: DuckDB over Pandas/Polars
#39Why not both? https://ibis-project.org/
Ibis looks very promising.
There are also other ways to use both.
https://duckdb.org/docs/guides/python/polars.html
All of this dataframe compatibility is awesome. (much thanks to Arrow and others)
Re: DuckDB over Pandas/Polars
#40My biggest issue with DuckDB is its not willing to implement edits to blob storages which allow edits (Azure). Having common object/blob storages that can be interacted and operated by multiple process will make it much more amenable to many data science driven workflows.
Probably not exactly what you mean or asking for, but the work Motherduck is doing looks promising.
https://motherduck.com/blog/differential-storage-building-bl...
Hopefully it finds its way into duckdb's repo some day.