Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB over Pandas/Polars

pgrs.net

Post reply on HN
Print view View original

41–43 of 43 posts

Re: DuckDB over Pandas/Polars

#41

Post by chrisjc »

wenc wrote: post #25
jgalt212 wrote: post #22

At what database size does it make sense to move from SQLite to DuckDB? My use case is off-line data analysis, not query / response web app.

It's not so much about size but about usage pattern. If your workloads require fast writes and reads, SQLite will probably work fine. If you're looking to run analytic, columnar queries (which tend to involve a lot of aggregation and joins on a few columns (say less than 50) at a time), then DuckDB is way more optimized. Oversimplifying, Sqlite is more OLTP and DuckDB is more OLAP.

Probably also worth mentioning that DuckDB can interact with SQLite dbs.

https://duckdb.org/docs/extensions/sqlite.html https://duckdb.org/docs/guides/database_integration/sqlite.h...

Thus potentially making duckdb an HTAP-like option.

Re: DuckDB over Pandas/Polars

#42

Post by Incipient »

swasheck wrote: post #34
Incipient wrote: post #27

Earlier quoted context omitted.

I'm curious how much simpler to build, manage, and run vs cost it would be to simply running a database on a large vultr/DO instance and paying for 2tb of storage? I feel like you'd get away with the whole thing for around $500/mo depending on how much compute was needed?

well that's not the infrastructure we have. we are primarily an aws shop so we use the resources available to us in the context of our infrastructure decisions. it would be a hard sell to buy something outside of that ecosystem.

I understand that's the infrastructure you have. But that's more describing vendor lock-in haha.

Most of my work is with clients that don't have any set infrastructure yet, so was curious if anyone had any anecdotes.

Re: DuckDB over Pandas/Polars

#43

Post by ramraj07 »

szarnyasg wrote: post #33
ramraj07 wrote: post #15

Earlier quoted context omitted.

Local drives. DONT USE EBS! you’ll incur a huge IO charge. You have to choose instances with attached nvme storage which means one of the storage optimized instances. Reading the data off s3 will mean you will be slower than offerings like snowflake. Snowflake has optimized the crap out of doing analytics in s3, so you can’t beat it with something as simple as duckdb. Importantly you need the data in some distributed…

Hi – DuckDB Labs devrel here. It's great that you find DuckDB useful! On the setup side, I agree that local (instance-attached) disks should be preferred but does EBS incur an IO fee? It incurs a significant latency for sure but it doesn't have a per-operation pricing: > I/O is included in the price of the volumes, so you pay only for each GB of storage you provision. ( https://aws.amazon.com/ebs/pricing/ )

Can’t remember anymore, but it’s either (a) the gp2 volumes were way too slow for the ops or (b) the IOPs charges made it bad. To be clear I didn’t do it on duckdb but hosted a Postgres. I moved to light sail instead and was happy with it (you don’t get attached SSD in ec2 until you go to instances that are super large).
Post reply on HN