11–20 of 208 posts
Yet this benchmark is at the top of HN for some reason.
If I asserted: "It's faster to put 10,000 rows of csv data in single file instead of 10,000 individual files" even the most junior programmer would likely say "Well, duh, it's 1 file instead of 10,000". Yet this benchmark is at the top of HN for some reason.
I for one am interested in the result and the reason.
If I asserted: "It's faster to put 10,000 rows of csv data in single file instead of 10,000 individual files" even the most junior programmer would likely say "Well, duh, it's 1 file instead of 10,000". Yet this benchmark is at the top of HN for some reason.
This is weird benchmarketing. They are comparing reading/writing 100,000 individual files vs writing 100,000 entries into a single file(sqlite database). For comparison one could concatenate the same data into a single big file even faster than into sqlite. They then do not offer logical analysis as to why things are faster. My understanding is that reads are probably faster due to operating system readahead being ab…
> For comparison one could concatenate the same data into a single big file even faster than into sqlite. One could, but then one would have great difficulty retrieving the individual files back when needed. The point is not about what has the greatest raw speed, the point is that that for applications that read lots of small files from the filesystem, they'll possibly get better performance and almost certainly use…
Not that great, and the point is its comparing apples to oranges, pointless.
If I asserted: "It's faster to put 10,000 rows of csv data in single file instead of 10,000 individual files" even the most junior programmer would likely say "Well, duh, it's 1 file instead of 10,000". Yet this benchmark is at the top of HN for some reason.
"Duh, I always knew that" is easy to say when you don't have to provide proof.
Also, are you sure that CSV will be faster as well? Do you have a benchmark for it?
If I asserted: "It's faster to put 10,000 rows of csv data in single file instead of 10,000 individual files" even the most junior programmer would likely say "Well, duh, it's 1 file instead of 10,000". Yet this benchmark is at the top of HN for some reason.
Your intuition far surpasses mine. I for one am interested in the result and the reason.
Compare to the typical case of adding another row by appending to an already allocated disk block. No seek operation needed.
This is weird benchmarketing. They are comparing reading/writing 100,000 individual files vs writing 100,000 entries into a single file(sqlite database). For comparison one could concatenate the same data into a single big file even faster than into sqlite. They then do not offer logical analysis as to why things are faster. My understanding is that reads are probably faster due to operating system readahead being ab…
https://eigenstate.org/paste/d41e3b69d258e3b52f7cee7a9921f02...
Earlier quoted context omitted.
> For comparison one could concatenate the same data into a single big file even faster than into sqlite. One could, but then one would have great difficulty retrieving the individual files back when needed. The point is not about what has the greatest raw speed, the point is that that for applications that read lots of small files from the filesystem, they'll possibly get better performance and almost certainly use…
> have great difficulty retrieving Not that great, and the point is its comparing apples to oranges, pointless.
You’d end up rewriting your own version of a database.
If I asserted: "It's faster to put 10,000 rows of csv data in single file instead of 10,000 individual files" even the most junior programmer would likely say "Well, duh, it's 1 file instead of 10,000". Yet this benchmark is at the top of HN for some reason.
So you never think that using database to store file will be slower than using file to store file? "Duh, I always knew that" is easy to say when you don't have to provide proof. Also, are you sure that CSV will be faster as well? Do you have a benchmark for it?