Leveraging SIMD: Splitting CSV Files at 3Gb/S
blog.tinybird.co
Leveraging SIMD: Splitting CSV Files at 3Gb/S
1–10 of 43 posts
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#2Pretty similar article from very recently: https://nullprogram.com/blog/2021/12/04/
Discussion: https://news.ycombinator.com/item?id=29439403
The article mentions in an addendum (and BeeOnRope also pointed it out in the HN thread) a nice CLMUL trick for dealing with quotes originally discovered by Geoff Langdale. That should work here for a nice speedup.
But without the CLMUL trick, I'd guess that the unaligned loads that generally occur after a vector containing both quotes and newlines in this version (the "else" case on lines 34-40) would hamper the performance somewhat, since it would eat up twice as much L1 cache bandwidth. I'd suggest dealing with the masks using bitwise operations in a loop, and letting i stay divisible by 16. Or just use CLMUL :)
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#3Nice, but I'm afraid real world CSVs are a lot more complicated than described so don't use this code in production.
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#4Nice, but I'm afraid real world CSVs are a lot more complicated than described so don't use this code in production.
If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format () it can definitely make sense to use a high-speed optimized ingester.
One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which means any form of SIMD optimization is right off the table (okay, maybe something could be done in Java, but it seems incredibly complex, see https://www.morling.dev/blog/fizzbuzz-simd-style/):
- PHP isn't optimized anywhere, but at least it's C: https://github.com/php/php-src/blob/1c0e613cf1a24cdc159861e4...
- Python's C implementation is the same: https://github.com/python/cpython/blob/main/Modules/_csv.c
- Java doesn't have a "standard" way at all (https://www.baeldung.com/java-csv-file-array), and OpenCSV seems the usual object-oriented hell (https://sourceforge.net/p/opencsv/source/ci/master/tree/src/...).
- Ruby's CSV is native Ruby: https://github.com/ruby/ruby/blob/bd65757f394255ceeb2c958e87...
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#5Nice, but I'm afraid real world CSVs are a lot more complicated than described so don't use this code in production.
If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format ( ) it can definitely make sense to use a high-speed optimized ingester. One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which me…
Python's csv imports _csv for core functionality, which is C: https://github.com/python/cpython/blob/main/Modules/_csv.c
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#6Why is the unit expression in topic messed up?
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#7Not sure how the author of this entry on HN managed to change original title from
gigabytes per second
to
gigabits per siemens
:)
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#8Nice, but I'm afraid real world CSVs are a lot more complicated than described so don't use this code in production.
If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format ( ) it can definitely make sense to use a high-speed optimized ingester. One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which me…
[deleted]
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#9Nice, but I'm afraid real world CSVs are a lot more complicated than described so don't use this code in production.
If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format ( ) it can definitely make sense to use a high-speed optimized ingester. One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which me…
It’s funny, csv files are so common and yet many mainstream languages don’t even attempt a decent parser baked in. I think dotnet has 3-4 different ones and as I recall they’re all pretty slow.
Re: Leveraging SIMD: Splitting CSV Files at 3Gb/S
#10Earlier quoted context omitted.
If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format ( ) it can definitely make sense to use a high-speed optimized ingester. One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which me…
Python's csv imports _csv for core functionality, which is C: https://github.com/python/cpython/blob/main/Modules/_csv.c
Thanks! Updated accordingly.