[deleted]
Choose DuckDB rather than SQLite
21–30 of 65 posts
Re: Choose DuckDB rather than SQLite
#22I ran this through an AI checker and it flagged half of it immediately. @dang, I know Substack just enabled Pangram integration, is there anyway you could get Y Combinator to spring for a Pangram subscription for the front page or something ?
Do you think venture capital firms are made of money?
well played sir. well played.
Re: Choose DuckDB rather than SQLite
#23AI slop. The content might be valuable but the framing makes it too painful to read. Please, folks, write with your own voice -- especially if it's for your business blog . It's good for you as an author (practice makes perfect) and it's good for your readers (whom you want to influence).
Agreed. It felt like I was reading a chat with Claude Code.
Re: Choose DuckDB rather than SQLite
#24[deleted]
Re: Choose DuckDB rather than SQLite
#25How are these two DB engines even comparable other than at the edges? They handle two completely separate workload types: one is more a general-purpose DB engine and the other is specifically for columnar datasets, analytics and the like -- of course a hand-tuned DB engine is going to destroy SQLite on any reasonable benchmark: SQLite wouldn't be optimized for that hand-tuned use-case whereas something like DuckDB is…
SQLite is _the_ tool of choice for local SQL databases with minimal overhead. If you needed a single file DB for an OLAP workload, SQLite was still the best option even if the technology wasn't an ideal fit. Duckdb is exciting specifically because SQLite/duckdb aren't comparable; we can stop shoehorning OLAP into an OLTP database.
I ran into this myself; I tried using SQLite to store the results of whole-internet rDNS scan and a count() over the entire DB could take 8 minutes. I used the wrong DB for the job and the narrative around SQLite/duckdb is around reckoning with perfectly reasonable limitations and tradeoffs that SQLite made.
Re: Choose DuckDB rather than SQLite
#26This is a bait and switch article. Compare apples and oranges, "oh btw look at our product".
Re: Choose DuckDB rather than SQLite
#27Despite its obvious advantages, the biggest drawback of DuckDB is its concurrency model [1]. If a process opens a database in read-write mode, it acquires an exclusive lock on the file. This prevents even simple read operations from other processes as long as the writer remains open. Maybe there's a simple workaround I haven't come across, but I found it to be quite a productivity killer. So yes, all these benchmarks…
Quack is now kind of a workaround for that limitation, as you can have a process with the lock offer read access to other processes. It's not perfect, but for that specific use case it's pretty good.
Re: Choose DuckDB rather than SQLite
#28Despite its obvious advantages, the biggest drawback of DuckDB is its concurrency model [1]. If a process opens a database in read-write mode, it acquires an exclusive lock on the file. This prevents even simple read operations from other processes as long as the writer remains open. Maybe there's a simple workaround I haven't come across, but I found it to be quite a productivity killer. So yes, all these benchmarks…
Duckdb has a server mode[1] which might alleviate some of those pain points. SQLite is a bit more precise in that only a single connection can write to the DB which provides more concurrency but still has pain points. For a single file DB either choice seems justifiable to manage complexity.
Re: Choose DuckDB rather than SQLite
#29> DuckDB's columnar engine That is workload specific. Title should be "Choose DuckDB rather than SQLite for Analytics" IMHO
This. The title might as well be “Choose a hammer rather than a wrench (when driving nails)”
Re: Choose DuckDB rather than SQLite
#30> DuckDB's columnar engine That is workload specific. Title should be "Choose DuckDB rather than SQLite for Analytics" IMHO
Yes - it's a specific workload for sure. SQLite is still the GOAT when it comes to OLTP, but DuckDB is really becoming the GOAT in the OLAP world - I think DuckDB is simply amazing and truly an amazing piece of technology for anyone working with large amounts of data.
> SQLite is still the GOAT when it comes to OLTP, but DuckDB is really becoming the GOAT in the OLAP world
Can you explain this more, especially why SQLite is best at OLTP and what happens at scale?