Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

Post reply on HN
Print view View original

191–200 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#191

Post by dagw »

mlthoughts2018 wrote: post #159
StefanKarpinski wrote: post #114

Earlier quoted context omitted.

Interestingly, CSV.jl can do all of those. This seems like a classic case of lack of composability in Python: you need a different CSV parser for each and every application even though the core logic of parsing the CSV format is the same. This is because anything fast has to be implemented in very concrete, specialized C code. In Julia, you can write something generic and let the compiler specialize it for you, so yo…

> “ In Julia, you can write something generic and let the compiler specialize it for you, so you get many different ways of applying the same thing for free.” Python also lets you easily achieve this in many ways - notably fused typing in Cython.

Just adding types doesn't give you that much extra performance in cython (2-3x at most in most cases in my experience) if you want 10x+ speed up you generally have to rewrite your code in a more 'cythonic' way.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#192

Post by cbkeller »

coldtea wrote: post #190
StefanKarpinski wrote: post #99

Earlier quoted context omitted.

You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Regarding your specific points: 1. Yes, it's possible to rig up some way of parsing in parallel in Python, like splitting the CSV file into multiple files and then parsing them in…

> You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Well, in Python typically one would use pandas "to parse a CSV file into a data frame", not the standard csv parser (which doesn't deal with data frames anyway), no?

That’s why Pandas read_csv is what they benchmarked

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#193

Post by bleepblorp »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

From my perspective, the fact that so much of computing has consolidated around a language that cannot support even the most rudimentary code completion or static analysis tools is deeply unfortunate. As a result of Python's mass adoption, uncountable thousands of hours of developer time have been wasted hunting down bugs that would have been caught by tooling in any statically-typed language.

Unless Python can fix this, by adopting static typing, it's not entirely fit for purpose in the ecosystem niche it has come to dominate. Replacing Python with a better language would be a net productivity boon for developers and scientists almost everywhere.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#194

Post by coldtea »

cbkeller wrote: post #192
coldtea wrote: post #190

Earlier quoted context omitted.

> You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Well, in Python typically one would use pandas "to parse a CSV file into a data frame", not the standard csv parser (which doesn't deal with data frames anyway), no?

That’s why Pandas read_csv is what they benchmarked

Ah, alrighty then.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#195

Post by cbkeller »

fermienrico wrote: post #187
ViralBShah wrote: post #181

Earlier quoted context omitted.

Nobody is spinning anything here. Julia's compilation time is a known quantity. See my comment in https://news.ycombinator.com/item?id=24750559 about the different phases of Julia's compilation and execution (from a user's perspective). Obviously, if you are not working on a problem that doesn't need Julia's speed (and makes it worth paying the compilation cost), and you are more comfortable with a different tool, yo…

Thanks for the insight. I think still for most engineering development - its mostly the one-off launch time that predominates. This is exactly why I don't use C++ for general purpose use. There is something to be said about how easy Python makes development look - despite of the package management warts. Btw, is there a reason why packages can't be precompiled and even distributed? I am sure you guys have thought abo…

> precompiled and even distributed

There is work on this. This can be done with https://julialang.github.io/PackageCompiler.jl/dev/, but so far only to a rather large binary / “bundle”

IIUC, In principle it should be possible to do much better if you knew for sure at compile time which methods you would need to dispatch to for the data you ultimately want to run on.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#196

Post by anthony_doan »

karbarcca wrote: post #26
CJefferson wrote: post #6

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues…

As the primary author of CSV.jl, I can help clarify on the posted issues: * #720: ended up not being an issue at all, but a misconfigured environment * #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug * #749, #734 are related to a new "beta" feature (CSV.Chunks) which…

> Compared with, say, data.table in R, or pandas in python, one of the things I enjoy most about Julia packages is that they're almost exclusively written in pure Julia.

R is going to eventually use Julia in their package. It's one of the gluest language I've ever learned so far.

It's actually R's strength. It doesn't have to compete when it can assimilate stuff in the backend while leveraging their existing user. Just look at STAN.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#197

Post by jakobnissen »

mlthoughts2018 wrote: post #103
nextos wrote: post #96

Earlier quoted context omitted.

I would also say, for those not familiar with Julia, that the main advantage of Julia vs Python or R is that you can write performant code in Julia that will be fast enough for most scenarios. For example, there are efforts to write a pure BLAS in Julia that is still performant [1]. If you are into numerical computing, you will quickly understand this is crazy cool. A consequence of that is composability. Most librar…

This is also very easy to achieve in Python by using Cython to selectively optimize code. That way for the 99% of code where such optimizations are meaningless and their requirement for static typing is a liability, you can ignore them, and only focus on the 1% of use cases where it matters.

... but Cython is not Python, is it? The argument that "Python can do it, you just have to use something other than Python) is very weird to me.

I began looking into Julia because I was forced into Cython during a project and I loathed having to add a separate, statically compiled section of my code that did not integrate well with the rest of Python (e.g. no stack traces through Cython), and kept breaking (because of some linker issues in Conda)

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#198

Post by RobinL »

boxed wrote: post #188
RobinL wrote: post #40

If you want to read csvs fast in Python, you should consider using the Apache Arrow[0] csv reader. Depending on your number of CPU cores it can be 10x-20x as fast as the native pandas reader. [1] More broadly, because Arrow is cross platform it can give you similar performance in many languages. And once the dataframe is in memory, you can share it between languages with no need for serialisation and deserialisation.…

Pandas is pretty slow and since it loads into memory it can be totally infeasible for even relatively small data sets. The csv module it what one should compare it to imo.

Apache Arrow reads csvs into memory in Arrow format, not pandas format. They are independent libraries that do different things. In Arrow, it is possible to read a csv in batches, obviating memory problems. See 'incremental reading' - http://arrow.apache.org/docs/python/csv.html

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#199

Post by [deleted] »

fermienrico wrote: post #187
ViralBShah wrote: post #181

Earlier quoted context omitted.

Nobody is spinning anything here. Julia's compilation time is a known quantity. See my comment in https://news.ycombinator.com/item?id=24750559 about the different phases of Julia's compilation and execution (from a user's perspective). Obviously, if you are not working on a problem that doesn't need Julia's speed (and makes it worth paying the compilation cost), and you are more comfortable with a different tool, yo…

Thanks for the insight. I think still for most engineering development - its mostly the one-off launch time that predominates. This is exactly why I don't use C++ for general purpose use. There is something to be said about how easy Python makes development look - despite of the package management warts. Btw, is there a reason why packages can't be precompiled and even distributed? I am sure you guys have thought abo…

[deleted]

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#200

Post by jakobnissen »

bleepblorp wrote: post #193
bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

From my perspective, the fact that so much of computing has consolidated around a language that cannot support even the most rudimentary code completion or static analysis tools is deeply unfortunate. As a result of Python's mass adoption, uncountable thousands of hours of developer time have been wasted hunting down bugs that would have been caught by tooling in any statically-typed language. Unless Python can fix t…

I think you're hitting on the main pain point of Julia. My bet is that we'll see strong typing dominate in the coming decade, with assistance from linters/static analysis becoming default.

Unfortunately, this is not too easy to create using Julia. Much easier than Python, sure, but not too easy. Tellingly, the Julia community apparently does not care for static analysis and have not begun to develop the capacity for it. As a consequence, much Julia code is written in a way that is un-analyzable, which digs it deeper into weak typing.

Post reply on HN