Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

21–30 of 165 posts

Re: DuckDB Internals Part 1

#21

Post by steve_adams_86 »

edweis wrote: post #7
jkubicek wrote: post #4

Earlier quoted context omitted.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

I personally find it useful to search logs with AI

Yes, it’s amazing for giving rails and structure to data so you can be sure an LLM is making more sense than it might with grep and jq. It also allows a little more sanity at scale with jobs like this. You can get pretty crazy with parquet in S3 with an engine like duckdb. And it’s dirt cheap to keep that stuff hanging around for future reference and sanity checking your understanding of things.

For data I reference frequently, and especially which I know will grow over time, I’ve started using Rill because it makes ad-hoc exploration very smooth and low-friction.

My process tends to be something like:

1. Explore logs or some other at least somewhat structured dataset

2. Use Claude to find useful patterns and determine how I might benefit from this data in ways I wasn’t yet aware

3. See how often it’s useful for decision making

4. If it’s frequently useful, formalize it as a view in my Rill instance and refine the models to maximize their utility

Re: DuckDB Internals Part 1

#22

Post by medvezhenok »

blackoil wrote: post #15
codingbear wrote: post #10

duckdb is so nice coupled with claude code. It extensive file support and some very interesting decisions on local caching data (like from S3 or snowflake) makes it easy to slice and dice almost any kind of tabular data.

> duckdb is so nice coupled with claude code Can you expand upon it? You mean claude code use it to store its memory/state or it can do business queries using DuckDB.

Claude code can write exploratory queries for you to give you a quick rundown on the shape of the data set, frequencies, missing values, etc etc (without having to load it into a more persistent data store or writing custom python scripts). I also find SQL snippets inherently more re-usable than custom python code.

You can also write a skill that CC can re-use if you're analyzing a lot of similar data sets with minor variance.

Re: DuckDB Internals Part 1

#23

Post by orthoxerox »

jkubicek wrote: post #4
steve_adams_86 wrote: post #2

> DuckDB has received widespread adoption because it's just so damn easy to use. This was a major factor in my initial adoption. Since then it has stuck because it’s also absurdly capable, versatile, and fast. If it wasn’t so easy to use I suspect I wouldn’t have adopted it when I did. The ergonomics are crazy. It still impresses me regularly.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

All kinds of data processing. For example, you download a million rows of metrics and load them in Excel to build pivot tables. It works, but now it's a billion rows. If you know SQL, it's a snap to point DuckDB at the source CSV or JSON and get the results in a second.

Re: DuckDB Internals Part 1

#25

Post by pknerd »

FTA:

> ..In-process means there's no server. You don't connect to DuckDB; you load it as a library inside your program, the same way you'd load NumPy or Polars

Does it mean it can perform all statistical computations as well if I want to use for algo trading?

Re: DuckDB Internals Part 1

#26

Post by Panzerschrek »

If DuckDB is so fast and has no data transfer overheads, does it need all this typical SQL machinery with filtering and joining via SELECT queries? Wouldn't it be simpler and faster to return all data to the caller code (all table rows, but only requested columns) and let it perform all other necessary data processing logic?

Re: DuckDB Internals Part 1

#27

Post by medvezhenok »

anitil wrote: post #5

DuckDb makes so much of my life easier, though I've never used it for large problems. The ability to run `select * from 'data.json'` is just lovely. The fact that it's also a powerhouse is so impressive, I'd usually expect a project to be good at small problems (like mine) xor large problems, but not both

Yup. And an extra benefit that you can treat any file like a table, so you can also do something like

  UPDATE my_table
  SET x = file1.x,
      y = file2.y
  FROM 'first_file.csv' file1
  LEFT JOIN 's3://my_bucket/second_file.parquet' file2
    ON file1.id = file2.id
  WHERE mytable.id = file1.id;

Re: DuckDB Internals Part 1

#28

Post by steve_adams_86 »

jkubicek wrote: post #4
steve_adams_86 wrote: post #2

> DuckDB has received widespread adoption because it's just so damn easy to use. This was a major factor in my initial adoption. Since then it has stuck because it’s also absurdly capable, versatile, and fast. If it wasn’t so easy to use I suspect I wouldn’t have adopted it when I did. The ergonomics are crazy. It still impresses me regularly.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

The most interesting use case lately has been using it as the transformation and validation engine for a CLI that handles scientific data. Some datasets are small and could have been handled at the application layer, but some are quite massive (especially genomic data). DuckDB bundles with the CLI and travels around any platform, is super lightweight, allows for easily running in CI, on a user’s machine, against datasets of all sizes, and so on.

There are other embeddable options out there but I found DuckDb fit better for the potentially massive datasets, and also because of how naturally it ingests the types of data we work with, some of its unique features, and how trivial it was to learn and integrate with the project.

Otherwise I use it almost daily for doing guardrailed data exploration with LLMs. I prefer SQL over random DSLs in AWS or Sentry or what have you. I’ll ingest the data I need and just run SQL against it. I mentioned in another comment that I’ll tend to store more useful data (especially data I export routinely, like infra cost reports) on S3 and use a Rill instance to do basic exploration in a GUI (it will query remote parquet files).

Re: DuckDB Internals Part 1

#29

Post by pknerd »

smithclay wrote: post #9

If you're reading this and curious: consider writing a duckdb community extension* or contributing to an existing one* duckdb is becoming a kind of data superglue between a lot of data ecosystems (GIS, observability, analytics, lakehouses, object storage, etc) that don't talk to each other typically, and it's worth checking out in 2026. * https://github.com/duckdb/extension-template * https://duckdb.org/community_ext…

Just curious whether one can earn money making these exts?
Post reply on HN