Frankly, all these implementations are disappointingly slow. Daniel Lemire and I wrote simdjson to use SIMD to read JSON quickly - CSV is strictly easier than that. I made a start on simdcsv but got bored, but the same principles would apply. IMO this task should be doable at 1-2GB/s on a single core; it's not something that should really need multiple cores.
CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
151–160 of 236 posts
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#152Earlier quoted context omitted.
We need more decades-stable stuff similar to LAPACK and BLAS, but for one step higher-level things. Just imagine if people had to rewrite all that numerical computation for every new language. Leaving some performance critical stuff in C is okay. We can expose them to a variety of languages then. > away from Theano [...] I had to switch from Tensorflow 1 to Tensorflow 2 I feel you. But I think this has more to do wit…
One of the neighboring comments mentioned something about changing assumptions. This is what limits the creation of decades-stable stuff. If you leave the performance crucial stuff in C, that stuff will not operate with the novel code you wrote (autodiff, probabilistic programming, GPU/TPU hardware, etc). Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actua…
>The actual reason these pure-Julia packages excite me is that they are generic enough to run relatively easily on different hardware (GPUs for instance)
Only if they were specifically written for GPUs. GPUs are incredibly limited. Expecting to run arbitrary code on them is only going to disappoint you. If you could just run regular code on GPUs then everyone would have done so. Programming language support is not a barrier. The GPU itself is. In fact, you should consider yourself lucky if you have problems that don't need GPUs.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#153The vroom R package is likely the fastest R package in read speed, not the ones used in this article. vroom was 13.1x faster than R's fread/data.table based on the read performance benchmark in https://cran.r-project.org/web/packages/vroom/vignettes/benc... so may be similar or faster in read speed vs julia.
Also Vroom never Worked For mE
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#154Earlier quoted context omitted.
I feel like you considering Python as having few real pain points in data science as either lack of knowledge of other languages or imagination/ambition. I do work on Python for data science/engineering in a production environment and I do find many of those all the time. Python does not feel like pouring my thoughts because I cannot write Python directly without taking hours to handle a few million points of data, s…
> so I have to juggle with multiple dialects (pandas, numpy, pytorch, tensorflow) NumPy should be enough for general computation. If you need autodiff or GPU then add in PyTorch. Pandas is more about various metadata than the actual numerical computing. If you want those types of features, the complexity doesn't disappear if you go to a different language. There's an effect where a new generation of developers see co…
I'll have to disagree with your defeatist generalization here: creating something with better knowledge of the problem and a lot of hindsight will not necessarily get as complicated for the same feature set. In fact the resets are probably necessary so you can get to even higher feature sets before getting so obtuse that it's a pain to move another step.
Julia doesn't even need full parity with numpy because you can trivially write your needs in straightforward Julia (in fact Julia does not have numpy, only Julia arrays). And Pytorch equivalent? Also uses Julia arrays and straightforward methods (as you can just differentiate Julia code directly). Pandas equivalent? It's a wrapper over Julia arrays. Need named tensors? Just pick another wrapper over Julia arrays and it will work on Julia's Pytorch equivalent without changing anything.
I do have to deal with Jupyter Notebooks, but they do have a lot of "real pain points" as well (it's always a pain when a data scientist gives one to add for deployment, half the time it does not work because it does not keep track of the whole execution history and need a major rewrite), but the main issue is the disconnect between the exploratory code and the production code. Julia's workflow (Revise + REPL) means I still write the structured code as usual, but every change in the code reflects in the REPL, like I'm interacting with the production code directly (and the Julia language supports a lot of commands to interface with the compiler, from finding source code to dumping the full structure of any object and even all types inferred). And of course, Julia also has Jupyter in the first place as it's name suggests, and a very cool alternative for some workflows like Pluto.jl that improves on reproducibility).
You might also want to try Erlang/Elixir, it's really amazing how you can trivially spawn hundreds of thousands of tasks that are incredibly robust even though it's a dynamic language (since you add not only tasks to do something but also to monitor and handle errors). And you can even connect and hot swap parts of the application live (as an example of superior interactivity).
I'm not saying that Python is not good, it wouldn't get where it is if it weren't. I'm saying that things can be way better, and we already have examples of it in most particular areas, but none that covers all of Python's strengths, which is why it will still be king for the foreseeable future. But I can only hope that we will get better and better tools to tackle more and more complex problems in the most simple and optimal way that we can given all that we learned as a community.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#155Earlier quoted context omitted.
I installed CSV.jl and tried it out on the insurance dataset from here " https://support.spatialkey.com/spatialkey-sample-csv-data/ ". First import took >10s just to precompile and then CSV.File took 5.21s to read the file. A fresh pandas install takes ~10s to import and then reads this file in 81.2 ms ± 1.25 ms. CSV.jl takes 38 ms on a second run so it is twice as fast, however especially to someone that just starts…
I tried out the dataset you mention. I use the latest Julia 1.5.2 and CSV.jl 0.7.7 and load the CSV package in 1s, read the file first time in 5s and then in 0.03s the second time. The precompile time you mention is only for the very first time you download a new version of a package, which is then cached for subsequent sessions. Your generalization from a one time precompilation of the CSV.jl package to >100x slowdo…
So I want to plot the data I just loaded. Since I wrote the comment I had closed the Julia window (not to make a point, btw.)
julia> import CSV (takes ~1-2 seconds)
julia> data = CSV.File("/User/orbifold/Downloads/FL_insurance_sample.csv) (takes seconds again...)
I want to explore the data. I vaguely remember there is Gadfly to do that. For some reason it depends on FFTW and a whole bunch of other packages, but the dependency story in Python is equally insane, so whatever. As I'm typing this, I'm waiting for Gadfly to precompile.
julia> @time import Gadfly (104.460307 seconds)
python> import matplotlib.pyplot as plt (~10s if you are unlucky and it has to generate the font cache)
Maybe it's a good idea to do a scatter plot of two of the fields? Let's find out
julia> @time Gadfly.plot(x=data.eq_site_limit, y=data.hu_site_limit) 0.320751 seconds
but that number is not accurate, in fact it took so long for the browser to open the result, that I had time to check the documentation on backends and see whether I'm missing something.
python> %time plt.scatter(data['eq_site_limit'], data['hu_site_limit']) 41.9 ms, a bit more for the window to open
Turns out that is not a useful plot... maybe a histogram?
julia> @time plot(x=data.eq_site_limit, Geom.histogram(bincount=10)) 0.042802 seconds
It in fact again takes several seconds until the browser shows a plot.
python> %time plt.hist(data["eq_site_limit"]) ~100ms
and the window with the plot opens instantly.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#156I built a pagerank implementation in Julia many years ago. It was operating on an adjacency list with half a billion edges, and I had to resort to writing a custom reader to get reasonable read speeds over the csv. In case anyone is interested: https://github.com/purzelrakete/Pagerank.jl/blob/070a193826f...
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#157Earlier quoted context omitted.
Because it is not a bottleneck. Modern SSDs can read data at more than 500 megabytes per second, which, on a single thread gives you just about 6-8 clock cycles to process a single byte. That’s not a lot of CPU instructions so it will take quite a number of threads to handle this bandwidth.
The SSD in my current workstation tops out at 5GB/s sequential reads and 680kIOPS. That's a modestly high-end consumer M.2 drive, not enterprise exotica. SSDs are really, really fast.
Yeah Intel optane and the like will be even faster.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#158Earlier quoted context omitted.
> “ Interestingly, CSV.jl can do all of those.“ That sounds like a poorly designed library to me. Instead of doing one thing well, it forces users to consume something meant as a jack of all trades, which may not meet their use case. I wish such a thing wasn’t forced in a standard package - rather split into different ones. Then I can easily compose them according to my needs, instead of having the stdlib force feed…
Well, the promise of overhead-free multiple dispatch is that you don't have to choose between a general solution with poor performance and a specific solution with good performance. You can have all the specific solutions you need in one place and automatically dispatch to the right one without any extra effort. I know it sounds too good to be true, but my limited experience so far is that it really is true , with su…
Multiple dispatch based on static typing only solves a very narrow set of optimization problems - most complex trade offs that would lead to the need for eg multiple different third party csv readers are not addressable at all by multiple dispatch.
I’d also point out that with fused typing in Cython, it’s also trivially easy to get overhead-free multiple dispatch in Python too, and unlike Julia, this had the benefit that it can be a gradual trade off, where you only bother writing the Cython module with autogenerated multiple dispatch implementations based on different types for the ~1% of use cases where it has any impact on the application, and you get the benefit of super easy to develop, flexible Python for everything else, with none of the code liability that comes from describing multiple dispatch generics in the 99% of the code that does not benefit from it.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#159Earlier quoted context omitted.
I disagree completely. It’s perfectly good enough for non-specialized use cases, which is exactly right for a stdlib offering. For anything else, factor it out as separate third party options (even if maintained by PSF itself) so users only install what they need for their special application. One person may install something that’s blazingly fast. Someone else may install something because it can convert the data to…
Interestingly, CSV.jl can do all of those. This seems like a classic case of lack of composability in Python: you need a different CSV parser for each and every application even though the core logic of parsing the CSV format is the same. This is because anything fast has to be implemented in very concrete, specialized C code. In Julia, you can write something generic and let the compiler specialize it for you, so yo…
> “ In Julia, you can write something generic and let the compiler specialize it for you, so you get many different ways of applying the same thing for free.”
Python also lets you easily achieve this in many ways - notably fused typing in Cython.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#160Earlier quoted context omitted.
I tried out the dataset you mention. I use the latest Julia 1.5.2 and CSV.jl 0.7.7 and load the CSV package in 1s, read the file first time in 5s and then in 0.03s the second time. The precompile time you mention is only for the very first time you download a new version of a package, which is then cached for subsequent sessions. Your generalization from a one time precompilation of the CSV.jl package to >100x slowdo…
So I want to plot the data I just loaded. Since I wrote the comment I had closed the Julia window (not to make a point, btw.) julia> import CSV (takes ~1-2 seconds) julia> data = CSV.File("/User/orbifold/Downloads/FL_insurance_sample.csv) (takes seconds again...) I want to explore the data. I vaguely remember there is Gadfly to do that. For some reason it depends on FFTW and a whole bunch of other packages, but the d…
Yes, Gadfly does have a large compile time. Reducing the time to first plot (essentially compilation time) has been a major focus for the 1.5 release.
People often use Plots.jl or PyPlot which have significantly lesser compile times.