Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

111–120 of 165 posts

Re: DuckDB Internals Part 1

#111

Post by aabdi »

lazarus01 wrote: post #92
gortok wrote: post #79

Earlier quoted context omitted.

> Like businesses don't care about the tool/tech itself, how do I find and approach them, and for which niche. You probably don't realize this, but you're asking one of the hardest questions when starting a business, and one of the questions others are least likely to be able to answer for you. "finding" a niche, and connecting to the business folks inside that 'niche' is hard, and is inherently a personal journey. T…

I use to be in sales before I became an MLE. There is a theory called diffusion of innovation. The simple explanation is that there are 5 different cohorts of buyers. Early adopters, visionaries, pragmatists, conservatives and laggards. Early adopters and visionaries are risk takers, who will make bold moves to achieve order of magnitude results. This is called the early market, which represents 13% of the market. Th…

Probably should tell guy to read the crossing the chasm book? Seems useful in this context.

Define the smallest market possible or something like that. I’m not sales though.

Re: DuckDB Internals Part 1

#112

Post by skotobaza »

romaniv wrote: post #91

It's an interesting project, but the discussion on HN looks weird. It gets brought up every few weeks[1] and everyone just spams comments with messages about how "fast" it is. DuckDB is fast for some specific workloads . If you use it for most other things, it is at least an order of magnitude slower than SQLite. It also has some limitations in terms of what SQL it will currently run (e.g. I immediately ran into an i…

> DuckDB is fast for some specific workloads

Yes, it's specifically promoted as DBMS for OLAP workload. And it's usually compared to ClickHouse, another analytical DBMS. So people who use it know why it's good.

Re: DuckDB Internals Part 1

#113

Post by skeeter2020 »

jkubicek wrote: post #4
steve_adams_86 wrote: post #2

> DuckDB has received widespread adoption because it's just so damn easy to use. This was a major factor in my initial adoption. Since then it has stuck because it’s also absurdly capable, versatile, and fast. If it wasn’t so easy to use I suspect I wouldn’t have adopted it when I did. The ergonomics are crazy. It still impresses me regularly.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

the taste that hooked me: the next time you have a bunch of json data, csvs or other data - local or remote - and someone wants some charts (for me it was "productivity" metrics from Jira combined with a bunch of other stuff). First it is very easy/fast to load this data; DuckDB has a very liberal parsing engine and good connectors. Second, I used to worry a lot about my table definitions and cleaning data before structuring it. Not anymore! With DuckDB I find myself iteratively transforming data and creating new tables, combining sources, converting columns, slicing/dicing/rotating. It's very easy to "remix" data and there are functions or extensions for everything you might want to do. There's so little friction to get started that I've found it just naturally becomes the multitool in my toolbox.

THis will give you some experience and you'll start to see applicable problem spaces for DuckDB in product areas, especially anything with BI or DW.

Re: DuckDB Internals Part 1

#114

Post by skeeter2020 »

raihansaputra wrote: post #65
jkubicek wrote: post #4

Earlier quoted context omitted.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

throwing in my 2 cents: It just replaced pandas for me. It's just so much easier to write sql against csv/json/whatever format data in jupyter/marimo notebooks through duckdb rather than reasoning through pandas. SQL is far more natural for me, and agents also work through it easily.

really learning SQL (syntax, boolean logic, how queries are broken down, etc) way back in uni has been the single biggest pay-off of my entire career.

Re: DuckDB Internals Part 1

#115

Post by skeeter2020 »

pedromlsreis wrote: post #95

DuckDB is a great example of how far you can get by removing unnecessary layers... Columnar layout and vectorized execution is a powerful combination for OLAP workloads.

the CSV parser is really good too. Anyone who's struggled with consuming CSV/TSV knows this is not trivial. DuckDB uses hueristics to be very liberal in what it accepts without crapping out like many parsers.

Re: DuckDB Internals Part 1

#116

Post by xtracto »

axegon_ wrote: post #83

I use duckdb HEAVILY at work and it's been a game changer. I'm sifting through terabytes of data multiple times a day, mixing, matching, updating, filtering, DuckDB is second to none. For anyone that hasn't used it: you are missing out.

This may be useful for somebody: We are also using DuckDB heavily at my workplace (we do Tax analytics of very large companies with huge amounts of data). We have certain DuckDB processes that happened in AWS infrastructure, where the data is saved in GP3 disks.

We didn't know that for GP3 disks, you can increase not only IOPS but also Read/Write Throughput [1] which by default is 125 MB/s. So by default we were not seeing the performance we expected.

Once we increased the throughput of the EBS, it was amazing. So if you are not seeing the performance you read about online when using DuckDB, it may be something like that.

[1] https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...

Re: DuckDB Internals Part 1

#117

Post by skeeter2020 »

mcv wrote: post #59

Is everything becoming columnar? Parquet stores data per column instead of per row because it improves compression. I get that. Arrow apparently is columnar, and now DuckDB also gets its efficiency by treating data as columns instead of rows? I still need to wrap my head around how that works, but it's a fascinating development.

compression is a side effect but not really the goal. To simplify, analytical queries often filter on a specific column value, and if these are laid out contiguously it makes disk-level reads much faster than rows that would involve read-skip-read-etc. In transactional systems data is typically written as rows though, so that's likely slower in a columnar system. As a general rule, heavy read workflows with known access patterns is going to benefit from a columnar layout.

Re: DuckDB Internals Part 1

#118

Post by tomjakubowski »

pierregillesl wrote: post #81
pknerd wrote: post #30

umm can we say it can replace SQLite?

I wouldn’t see it as a replacement. SQLite and DuckDB solve different problems and actually complement each other quite well. SQLite is excellent for transactional workloads (OLTP), while DuckDB shines for analytical workloads (OLAP), especially time-series data and aggregations. We’ve been using both side-by-side in an open-source project for about two years: SQLite for configuration and transactional data, DuckDB f…

You can even use DuckDB to query SQLite :^)

Re: DuckDB Internals Part 1

#119

Post by skeeter2020 »

pierregillesl wrote: post #81
pknerd wrote: post #30

umm can we say it can replace SQLite?

I wouldn’t see it as a replacement. SQLite and DuckDB solve different problems and actually complement each other quite well. SQLite is excellent for transactional workloads (OLTP), while DuckDB shines for analytical workloads (OLAP), especially time-series data and aggregations. We’ve been using both side-by-side in an open-source project for about two years: SQLite for configuration and transactional data, DuckDB f…

DuckDB kind of created this false comparison by their own early positioning, but I've tried to charitably interpret it as modeling the spirit and motivations of SQLite, not literally being "the SQLite for Analytics". Aside from both being in-process databases they are very different.

Re: DuckDB Internals Part 1

#120

Post by pants2 »

xtracto wrote: post #116
axegon_ wrote: post #83

I use duckdb HEAVILY at work and it's been a game changer. I'm sifting through terabytes of data multiple times a day, mixing, matching, updating, filtering, DuckDB is second to none. For anyone that hasn't used it: you are missing out.

This may be useful for somebody: We are also using DuckDB heavily at my workplace (we do Tax analytics of very large companies with huge amounts of data). We have certain DuckDB processes that happened in AWS infrastructure, where the data is saved in GP3 disks. We didn't know that for GP3 disks, you can increase not only IOPS but also Read/Write Throughput [1] which by default is 125 MB/s. So by default we were not…

This seems crazy low to me. AWS has default 3K IOPS and 125 MB/s throughput, meanwhile my Macbook Pro has 700K IOPS and 14.5GB/s throughput.

Is Amazon running on super outdated legacy networking?

Post reply on HN