ai; dr
DuckDB Internals Part 1
131–140 of 165 posts
Re: DuckDB Internals Part 1
#132I have a database on my local machine with 200 million records in one table and 2 other related tables. Even the most complex queries take about 1 minute to vibe code and paste into the DuckDB browser playground UI and always less than 5 seconds to return the result. For a humble product manager it feels like a superpower.
Re: DuckDB Internals Part 1
#133Earlier quoted context omitted.
I'm not convinced that is true. The JCB digger didn't put groundworkers out of business. A consultant that can get more done in a day is worth a lot more than one who can't. There is still skill required in wielding the tool of the day and that skill is marketable.
Sorry, I don't think your non-software analogy that existed before 2025 helps at all in this specific topic about software consultancy.
Meanwhile, people are in fact making actual money solving problems for others by building use-case-specific solutions. Perhaps you’re unaware of them, but they do exist. They prefer to operate off the radar.
Re: DuckDB Internals Part 1
#134If you're reading this and curious: consider writing a duckdb community extension* or contributing to an existing one* duckdb is becoming a kind of data superglue between a lot of data ecosystems (GIS, observability, analytics, lakehouses, object storage, etc) that don't talk to each other typically, and it's worth checking out in 2026. * https://github.com/duckdb/extension-template * https://duckdb.org/community_ext…
Just curious whether one can earn money making these exts?
Open source in general is notoriously difficult to easily monetize.
Re: DuckDB Internals Part 1
#135DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…
>Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Nice! How do you set things up so that your engineers's claude code sessions upload to S3? Thanks for the help in advance
We have a hook that runs on session start and session end that sends data to a lambda with a hard coded JWT token that we ship in the code
We added that to the managed settings for our Claude instance as a “base” plugin and provision it to all machines using JAMF
A non-enterprises version of that would be to add this hook in your main repository’s .claude folder
Re: DuckDB Internals Part 1
#136It's an interesting project, but the discussion on HN looks weird. It gets brought up every few weeks[1] and everyone just spams comments with messages about how "fast" it is. DuckDB is fast for some specific workloads . If you use it for most other things, it is at least an order of magnitude slower than SQLite. It also has some limitations in terms of what SQL it will currently run (e.g. I immediately ran into an i…
I’m sure the use of duckdb may seem weird for normal developers, but for data people it really is game-changing, especially for data scientists or business analysts.
Re: DuckDB Internals Part 1
#137Earlier quoted context omitted.
This seems crazy low to me. AWS has default 3K IOPS and 125 MB/s throughput, meanwhile my Macbook Pro has 700K IOPS and 14.5GB/s throughput. Is Amazon running on super outdated legacy networking?
SAN vs local. Local NVME (“instance storage“) on AWS is wicked fast too, but live and dies with the instance
in 2026 you can full the PCIe BW from remote storage at roughly the same latency as your local NVMe drive (hello NVMeoF).
your comment would have been valid 10 years ago, maybe.
Re: DuckDB Internals Part 1
#138It's an interesting project, but the discussion on HN looks weird. It gets brought up every few weeks[1] and everyone just spams comments with messages about how "fast" it is. DuckDB is fast for some specific workloads . If you use it for most other things, it is at least an order of magnitude slower than SQLite. It also has some limitations in terms of what SQL it will currently run (e.g. I immediately ran into an i…
The article has an explanation for what kind of database it is. After reading that one sentence you wouldn't write the second paragraph.
> it's optimized for the kind of queries that scan millions of rows to filter, aggregate, and join — not the kind that look up a single record by primary key
Re: DuckDB Internals Part 1
#139It's an interesting project, but the discussion on HN looks weird. It gets brought up every few weeks[1] and everyone just spams comments with messages about how "fast" it is. DuckDB is fast for some specific workloads . If you use it for most other things, it is at least an order of magnitude slower than SQLite. It also has some limitations in terms of what SQL it will currently run (e.g. I immediately ran into an i…
> DuckDB effectively appeared on the scene two years ago
I don't think so.
Re: DuckDB Internals Part 1
#140I use duckdb HEAVILY at work and it's been a game changer. I'm sifting through terabytes of data multiple times a day, mixing, matching, updating, filtering, DuckDB is second to none. For anyone that hasn't used it: you are missing out.
I asked Claude which tools it would most want to add to a machine we were working on. #1 was DuckDB.