Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

Post reply on HN
Print view View original

181–190 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#181

Post by ViralBShah »

fermienrico wrote: post #170
ViralBShah wrote: post #160

Earlier quoted context omitted.

Yes, Gadfly does have a large compile time. Reducing the time to first plot (essentially compilation time) has been a major focus for the 1.5 release. People often use Plots.jl or PyPlot which have significantly lesser compile times.

@ViralBShah and @StefanKarpinski I've had similar problems and so have many of my colleagues in silicon valley. No matter how you spin it and whatever benchmarks you show - Julia is a very slow fast language. :) The UX of Julia needs major work - everything is just slow . If you've used Python for 10 years and it is like running through molasses. Benchmarks are meaningless for the most part. This is what bothers me a…

Nobody is spinning anything here. Julia's compilation time is a known quantity. See my comment in https://news.ycombinator.com/item?id=24750559 about the different phases of Julia's compilation and execution (from a user's perspective).

Obviously, if you are not working on a problem that doesn't need Julia's speed (and makes it worth paying the compilation cost), and you are more comfortable with a different tool, you should use that. You may in fact be better off using Python, or shell scripting or even Excel.

Precompilation time in Julia is akin to `make` in a project with C code. You only compile the library once and use it repeatedly, until you update it. Is it a dealbreaker? Perhaps it is for you (assuming you are using precompilation in the right context). It is not for many who work with Julia day in and day out, and it is something that we continue to improve.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#182

Post by ViralBShah »

datanecdote wrote: post #172
ViralBShah wrote: post #146

Earlier quoted context omitted.

Perhaps not directly relevant to your point here, but thought it would be interesting to anyone following along. Jacob Quinn (karbacca) also has a Julia package for integrating Julia into the Arrow ecosystem: https://github.com/JuliaData/Arrow.jl

Thanks Viral. To be clear, I’m a python user who’s cheering for Julia, because I live the problems of python and do see the potential of Julia as a better path. But unfortunately I’m not prepared to be the early adopter (at least in my day job), and will wait until other, braver users have sanded off the rough edges. God speed and good luck.

That's a completely reasonable viewpoint. Many users of Julia and contributors start out experimenting with it and then end up bringing it into their work when they feel comfortable with it. I hope you will have the same experience one day.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#183

Post by newen »

UMetaGOMS wrote: post #147
krastanov wrote: post #46

Earlier quoted context omitted.

One of the neighboring comments mentioned something about changing assumptions. This is what limits the creation of decades-stable stuff. If you leave the performance crucial stuff in C, that stuff will not operate with the novel code you wrote (autodiff, probabilistic programming, GPU/TPU hardware, etc). Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actua…

> faster than many LAPACK/BLAS implementations IMO this is worth very little; as long as there is a heavily optimized version written in C/assembly (ie MKL), the fact that Julia might be faster than standard-but-slower versions like OpenBLAS doesn't actually gain me anything as a user.

Some people want to use their own types and number formats, so it can potentially be very useful.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#184

Post by dklend122 »

mlthoughts2018 wrote: post #158
cbkeller wrote: post #124

Earlier quoted context omitted.

Well, the promise of overhead-free multiple dispatch is that you don't have to choose between a general solution with poor performance and a specific solution with good performance. You can have all the specific solutions you need in one place and automatically dispatch to the right one without any extra effort. I know it sounds too good to be true, but my limited experience so far is that it really is true , with su…

Multiple dispatch based on static typing only solves a very narrow set of optimization problems - most complex trade offs that would lead to the need for eg multiple different third party csv readers are not addressable at all by multiple dispatch. I’d also point out that with fused typing in Cython, it’s also trivially easy to get overhead-free multiple dispatch in Python too, and unlike Julia, this had the benefit…

You are very misinformed here.

Julia doesn't have static typing and there are a very large set of problems that are trivially solved with it's type system.

See the tables.jl ecosystem for example

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#185

Post by dklend122 »

imtringued wrote: post #152
krastanov wrote: post #46

Earlier quoted context omitted.

One of the neighboring comments mentioned something about changing assumptions. This is what limits the creation of decades-stable stuff. If you leave the performance crucial stuff in C, that stuff will not operate with the novel code you wrote (autodiff, probabilistic programming, GPU/TPU hardware, etc). Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actua…

>The actual reason these pure-Julia packages excite me is that they are generic enough to run relatively easily on different hardware (GPUs for instance) Only if they were specifically written for GPUs. GPUs are incredibly limited. Expecting to run arbitrary code on them is only going to disappoint you. If you could just run regular code on GPUs then everyone would have done so. Programming language support is not a…

That's patent false. Julia has abstractions that make it very easy to write generic code .

Sure, no arbitrary loops though,..but a constrained subset will work (see kernel abstractions.jl)

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#186

Post by fermienrico »

throwaway894345 wrote: post #174
fermienrico wrote: post #92

Earlier quoted context omitted.

I find Rust really difficult to get into :( Python, you're right has a bunch of problems with datetime and packages, etc. but when it comes to productivity, it is insane how amazing Python is. Also being able to hack internals of python library and "batteries included" philosophy is why it is the most popular language in the world now overtaking Java.

Rust does have a steep learning curve and Python is a very productive language... until you are the one responsible for builds and deploys, or performance, or concurrency, or quality, or etc. I think you’ll find that Go hits a nice sweet spit between all of those competing concerns even if it isn’t the best at any of them (in general; however, probably not for data science in particular).

Thanks, I am willing to try both Rust and Go.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#187

Post by fermienrico »

ViralBShah wrote: post #181
fermienrico wrote: post #170

Earlier quoted context omitted.

@ViralBShah and @StefanKarpinski I've had similar problems and so have many of my colleagues in silicon valley. No matter how you spin it and whatever benchmarks you show - Julia is a very slow fast language. :) The UX of Julia needs major work - everything is just slow . If you've used Python for 10 years and it is like running through molasses. Benchmarks are meaningless for the most part. This is what bothers me a…

Nobody is spinning anything here. Julia's compilation time is a known quantity. See my comment in https://news.ycombinator.com/item?id=24750559 about the different phases of Julia's compilation and execution (from a user's perspective). Obviously, if you are not working on a problem that doesn't need Julia's speed (and makes it worth paying the compilation cost), and you are more comfortable with a different tool, yo…

Thanks for the insight. I think still for most engineering development - its mostly the one-off launch time that predominates. This is exactly why I don't use C++ for general purpose use.

There is something to be said about how easy Python makes development look - despite of the package management warts.

Btw, is there a reason why packages can't be precompiled and even distributed? I am sure you guys have thought about that.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#188

Post by boxed »

RobinL wrote: post #40

If you want to read csvs fast in Python, you should consider using the Apache Arrow[0] csv reader. Depending on your number of CPU cores it can be 10x-20x as fast as the native pandas reader. [1] More broadly, because Arrow is cross platform it can give you similar performance in many languages. And once the dataframe is in memory, you can share it between languages with no need for serialisation and deserialisation.…

Pandas is pretty slow and since it loads into memory it can be totally infeasible for even relatively small data sets. The csv module it what one should compare it to imo.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#190

Post by coldtea »

StefanKarpinski wrote: post #99
dmw_ng wrote: post #51

There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that…

You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Regarding your specific points: 1. Yes, it's possible to rig up some way of parsing in parallel in Python, like splitting the CSV file into multiple files and then parsing them in…

>You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame.

Well, in Python typically one would use pandas "to parse a CSV file into a data frame", not the standard csv parser (which doesn't deal with data frames anyway), no?

Post reply on HN