DuckDB – Data power tools for your laptop, now in Clojure (2023) • Hacker News Board

Re: DuckDB – Data power tools for your laptop, now in Clojure (2023)

#21

Post by walthamstow » Wed, Aug 05, 2026, 1:52 PM UTC

HackerThemAll wrote: post #17

DuckDB, for those who don't know it, has a great extension system, for example it can directly use OLTP databases such as PostgreSQL, MySQL, SQLite, SQL Server; cloud data warehouses/data lakes and big data formats (Iceberg, Delta, Snowflake, Hive, ORC, Parquet, AVRO), other data sources (ODBC), storage (S3), and much much more. https://duckdb.org/docs/current/core_extensions/overview https://duckdb.org/community_ext…

Nice tip, thanks. I may now make Duck CLI the sole gateway to all the databases I have to use at work.

Re: DuckDB – Data power tools for your laptop, now in Clojure (2023)

#22

Post by encoderer » Wed, Aug 05, 2026, 2:09 PM UTC

sethev wrote: post #19

reacharavindh wrote: post #18

Earlier quoted context omitted.

I’m confused as an old school storage guy. > self-hosted directly on NVMe backed by cheap and limitless object storage. How is that?! NVMe is a protocol for fast PCIE based local storage or NVMe fabric which is PCIe over network. Object storage(in the sense of S3, R2 etc) are usually networked horizontally scaling non posix bucketed storage. They are much much slower because of the network calls.. How would NVMe map…

I'm just guessing but observability is often looking at small slices of hot data and then there's a vast set of cold data that is occasionally needed. Sounds like NVMe is for the hot data cache and object storage is for cold data.

Yes, you nailed it. Old data is important sometimes, like when a problem has been identified and investigated, but most workloads are looking at the current state of the system. So we keep the hot data cached locally (not nas/ebs) and s3 is always the source of truth. DuckDB over parquet files on a local ssd is fast enough you don’t need a traditional database.

We are using DuckLake with a “lakehouse” architecture for the observability agent.

Re: DuckDB – Data power tools for your laptop, now in Clojure (2023)

#23

Post by didibus » Wed, Aug 05, 2026, 9:32 PM UTC

alex_smart wrote: post #15

Why would you need this over just using the jdbc driver?

> but it's irritating to perform a full row-to-column conversion through an inefficient, non-batched API in order to get the data through JDBC and into TMD

[...]

> In the last two years, DuckDB improved a lot. Importantly, the C interface now provides a batched system for both inserts and querying, which enables processing very large joins

That's why. Basically JDBC doesn't offer the batched APIs needed to get the same performance.

Re: DuckDB – Data power tools for your laptop, now in Clojure (2023)

#24

Post by HackerThemAll » Thu, Aug 06, 2026, 8:26 AM UTC

walthamstow wrote: post #21

HackerThemAll wrote: post #17

DuckDB, for those who don't know it, has a great extension system, for example it can directly use OLTP databases such as PostgreSQL, MySQL, SQLite, SQL Server; cloud data warehouses/data lakes and big data formats (Iceberg, Delta, Snowflake, Hive, ORC, Parquet, AVRO), other data sources (ODBC), storage (S3), and much much more. https://duckdb.org/docs/current/core_extensions/overview https://duckdb.org/community_ext…

Nice tip, thanks. I may now make Duck CLI the sole gateway to all the databases I have to use at work.

It will certainly allow you to automate tons of stuff, such as data movement between those various sources, data transformations (ETL/ELT) and even simple data conversions - such as from CSV to Parquet, or from ORC to JSON etc. etc. Happy quacking!