Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asynchronous I/O in DuckDB: Work, Thread, Work

duckdb.org

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Asynchronous I/O in DuckDB: Work, Thread, Work

#21

Post by hilariously »

pdet wrote: post #19
ilyagr wrote: post #16

Do the CSV files allow quoted newlines? If yes, what's the trick to avoid checking the whole file too find out whether a newline is quoted or a record separator when reading it from the middle in an async thread?

DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation. I've never gotten around to wri…

Burning my points to say dude that's sick, parsing CSVs is hell I am genuinely going to watch this thank you.

Re: Asynchronous I/O in DuckDB: Work, Thread, Work

#22

Post by Asmod4n »

diarrhea wrote: post #10

Does having the worker pool hold as many threads as cores work well alongside the async pool? It is basically oversubscribed by design. I built a system once which had (this is Rust) a Rayon worker thread pool of 4 threads and a Tokio async pool of 2 (multithreaded runtime). On a system of 6 vCPU. This ended up working fine. Tokio was not starved so handled network requests at low latency. One difference is DuckDB is…

Network IO is heavily NIC queue bound, if your NIC only has one queue it just makes it slower to do any threading workload against it.

On my ryzen 9 it needs around 8 cores to do the same work in a threaded io loop than you can do single threaded. And the mechanism doesnt matter, you could share an fd, use SO_REUSEPORT or just share memory between threads.

Just doing the sharing makes everything extremely slow. One context switch becomes more expensive than just doing it single threaded.

Re: Asynchronous I/O in DuckDB: Work, Thread, Work

#23

Post by mahogany »

pdet wrote: post #14
bburnett44 wrote: post #6

Using 512gb of ram for a 22gb remote file does feel a bit weird for a benchmark but maybe they couldn’t get a large number of cores without lots of memory?

The main reason I decided to use a beefier machine is that it gives me flexibility when benchmarking, without the need to set up different environments. The CSV data, for example, is >80 GB. We can also “scale down the machine” for experiments where we want to stress-test lower-memory scenarios or use fewer threads by configuring DuckDB’s settings (e.g., SET memory_limit = '10GB'; or SET threads TO 1;). (Disclaimer:…

Note that SET memory_limit is a soft limit and can be completely ignored for some tasks, so this wouldn’t be the same as having a limit on physical memory.

Re: Asynchronous I/O in DuckDB: Work, Thread, Work

#24

Post by pletnes »

pdet wrote: post #19
ilyagr wrote: post #16

Do the CSV files allow quoted newlines? If yes, what's the trick to avoid checking the whole file too find out whether a newline is quoted or a record separator when reading it from the middle in an async thread?

DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation. I've never gotten around to wri…

You’re officially my hero, the duckdb csv parser is the fastest I could find and helped save a data project with hundreds of gigabytes of csv pain.

Re: Asynchronous I/O in DuckDB: Work, Thread, Work

#26

Post by orthoxerox »

ilyagr wrote: post #16

Do the CSV files allow quoted newlines? If yes, what's the trick to avoid checking the whole file too find out whether a newline is quoted or a record separator when reading it from the middle in an async thread?

In general, yes. But Hive, for example, doesn't support embedded newlines or record separators that are not newlines, which means that a lot of older CSV files containing big data do not allow quoted newlines.

Re: Asynchronous I/O in DuckDB: Work, Thread, Work

#29

Post by _zoltan_ »

hilariously wrote: post #21
pdet wrote: post #19

Earlier quoted context omitted.

DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation. I've never gotten around to wri…

Burning my points to say dude that's sick, parsing CSVs is hell I am genuinely going to watch this thank you.

there isn't any upvote limit on HN.

Re: Asynchronous I/O in DuckDB: Work, Thread, Work

#30

Post by ilyagr »

pdet wrote: post #19
ilyagr wrote: post #16

Do the CSV files allow quoted newlines? If yes, what's the trick to avoid checking the whole file too find out whether a newline is quoted or a record separator when reading it from the middle in an async thread?

DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation. I've never gotten around to wri…

[flagged]
Post reply on HN