Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

91–100 of 165 posts

Re: DuckDB Internals Part 1

#91

Post by romaniv »

It's an interesting project, but the discussion on HN looks weird. It gets brought up every few weeks[1] and everyone just spams comments with messages about how "fast" it is.

DuckDB is fast for some specific workloads. If you use it for most other things, it is at least an order of magnitude slower than SQLite. It also has some limitations in terms of what SQL it will currently run (e.g. I immediately ran into an issue with recursive queries). That will probably get better with time.

[1] If you search HN for "sqlite" and "duckdb" you get 4,310 hits and 2,398 hits respectively. That's a very heavy skew, considering SQLite is everywhere and had been around for a quarter century, while DuckDB effectively appeared on the scene two years ago.

Re: DuckDB Internals Part 1

#92

Post by lazarus01 »

gortok wrote: post #79
pknerd wrote: post #68

Earlier quoted context omitted.

Thanks for your kind response. Could you guide further? Like businesses don't care about the tool/tech itself, how do I find and approach them, and for which niche. Thanks in Advance

> Like businesses don't care about the tool/tech itself, how do I find and approach them, and for which niche. You probably don't realize this, but you're asking one of the hardest questions when starting a business, and one of the questions others are least likely to be able to answer for you. "finding" a niche, and connecting to the business folks inside that 'niche' is hard, and is inherently a personal journey. T…

I use to be in sales before I became an MLE.

There is a theory called diffusion of innovation. The simple explanation is that there are 5 different cohorts of buyers. Early adopters, visionaries, pragmatists, conservatives and laggards. Early adopters and visionaries are risk takers, who will make bold moves to achieve order of magnitude results. This is called the early market, which represents 13% of the market. The pragmatists and conservatives make up the mainstream market which is about 70%.

In order to get into the mainstream market, you need solid adoption from the early market.

To choose a niche, you need to develop a solution that fits nicely into the buyers expectations for different types of market participants. There is the market alternative and product alternative. The market alternative is the solution that owns the highest proportion of market share. The product alternative is innovative tech that challenges superiority to the market alternative.

You need to introduce a solution that fits in between those participants to stand out.

To choose a solution, go to industry trade events and talk to people about high value problems that aren’t solved by current participants. That is the purpose of industry associations, to solve difficult problems.

Visionaries and early adopters love new vendors. They will champion you through their organization if your solution will help them meet their goals.

Good luck

Re: DuckDB Internals Part 1

#93

Post by squirrellous »

mcv wrote: post #59

Is everything becoming columnar? Parquet stores data per column instead of per row because it improves compression. I get that. Arrow apparently is columnar, and now DuckDB also gets its efficiency by treating data as columns instead of rows? I still need to wrap my head around how that works, but it's a fascinating development.

Those three things you mentioned kind of live in the same niche - offline data storage and querying. In that world yes everything has become columnar since it’s just better. Row-oriented is still the solution for online streaming use cases.

Re: DuckDB Internals Part 1

#94

Post by sceadu »

medvezhenok wrote: post #20
jkubicek wrote: post #4

Earlier quoted context omitted.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

Basically like a locally hosted Snowflake - it only shines if you have enough data to analyze (100 MB - 100 GB is probably the sweet-spot range - less than that and the benefits are small, more than that and you risk flying too close to the sun with memory usage). It has connectors for Postgres & other stores, so I find it faster to connect to a Postgres instance, pull all of the data from a table (even if the table…

In my experience it works OK with spilling to disk so I haven't had too much of a concern with memory usage... previously I had issues with it OOM'ing and failing (or maybe this was a skill issue?), but haven't had that happen recently.

Re: DuckDB Internals Part 1

#96

Post by tomjakubowski »

IshKebab wrote: post #55
refactor_master wrote: post #39

Earlier quoted context omitted.

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

That does look nicer if you have a Parquet file and want to analyze it. But DuckDB is also a database - if you want a persistent, reliable and mutable data store I don't think Polars would be suitable would it? (Genuine question - you sound like an expert and I'm not.)

The other thing DuckDB does quite well, and which is out of scope for polars, is its keeping of an absolute zoo of external data sources you can query with SQL from the same database client. It's an excellent data warehousing tool.

Re: DuckDB Internals Part 1

#97

Post by sceadu »

refactor_master wrote: post #39
holografix wrote: post #24

Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

I really hate SQL as a language for doing data-related tasks, but the core tech for it is often much better in terms of reliability (and is close to having performance parity IME)... specifically for working with larger-than-memory datasets. I think the team at Polars is working on improving larger-than-memory operations though.
Post reply on HN