Asynchronous I/O in DuckDB: Work, Thread, Work

Post by michaelmdresser » Sun, Aug 16, 2026, 3:36 PM UTC

mahogany wrote: post #23

pdet wrote: post #14

Earlier quoted context omitted.

The main reason I decided to use a beefier machine is that it gives me flexibility when benchmarking, without the need to set up different environments. The CSV data, for example, is >80 GB. We can also “scale down the machine” for experiments where we want to stress-test lower-memory scenarios or use fewer threads by configuring DuckDB’s settings (e.g., SET memory_limit = '10GB'; or SET threads TO 1;). (Disclaimer:…

Note that SET memory_limit is a soft limit and can be completely ignored for some tasks, so this wouldn’t be the same as having a limit on physical memory.

Post by quag » Sun, Aug 16, 2026, 10:04 PM UTC

pdet wrote: post #19

ilyagr wrote: post #16

Do the CSV files allow quoted newlines? If yes, what's the trick to avoid checking the whole file too find out whether a newline is quoted or a record separator when reading it from the middle in an async thread?

DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation. I've never gotten around to wri…

Post by mhh__ » Mon, Aug 17, 2026, 12:43 AM UTC

_zoltan_ wrote: post #28

datadrivenangel wrote: post #4

DuckDB is trending towards becoming a query engine, specifically the fastest analytical query engine. This is very good.

far from the fastest - our work on Presto on GPUs would like a word ;-)