Quack: The DuckDB Client-Server Protocol

I think one of the most common and elevating methods of using it has been combining disparate sources of data into multiple tables of a single instance so I can run queries locally and use DuckDB as a bridge between platforms.

Yesterday I pulled a bunch of data from Sentry, multiple log groups on AWS, and Github to figure out when some incidents occurred and how events correlated or caused each other.

Doing that in other tools is perfectly possible and fine, but the overhead of setting up a docker container or understanding requirements for setup or needing an account or whatever bespoke query language makes me lose interest immediately.

With this I only need to know SQL, optionally duckdb -ui, roughly how to ingest the sources correctly so they can be joined easily (in this case just make sure everything is a UTC time series), and I'm mostly off to the races. It works fine.

There are more sophisticated and cool and whatever ways to do this, but with Claude as an assistant you can do this with like 3.5 brain cells and get absolutely incredible results.

DuckDB is awesome partially because of how effortless it is and how little ceremony there is. Like SQLite, but even less friction. Having duckdb -ui as a little work bench is brilliant.

Highly suggest you take a look here: https://github.com/yjs/yjs

CRDT can absolutely do what you’re asking.

What drives such a high throughput difference between Quack and Arrow on high-volume operations ?

I'll try to search from source/Github, reply appreciated though, for example:

- when DuckDb bulk exports a table, does Quack benefit from pre-existing compression/encodings/0-copy where Arrow requires decode+re-encode ?

- the post mentions parallel reads, is the level of parallelism the same on Arrow vs Quack here ? Running the high throughput benchmark at resource saturation with increasing number of concurrent bulk-read clients would be more transparent

TypeScript: https://www.npmjs.com/package/@quack-protocol/sdk

Trivial deployment model, can avoid owning/managing your own server, easier coding in the sense that no-network means you can do normally-psychotic things like N+1 queries and single-row inserts and get away with it.

SQLite/DuckDB actually enables a bunch of normally-illegal behavior when you compare to normal databases. Backups is just copy&paste of a file; spamming queries willy-nilly becomes cheap; you can version the whole DB in git (can’t diff it properly.. but you can do cross-db queries with SQLite ATTACH); locking concerns goes out the window because it’s single-writer anyways.

But if I were actively trying to support multiple users with a single source of truth, I’d probably default to Postgres. If it’s single-user, default to SQLite/DuckDB. If it’s single-user with multiple devices, default to SQLite + replication.