Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

Post reply on HN
Print view View original

31–40 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#31

Post by CJefferson »

karbarcca wrote: post #26
CJefferson wrote: post #6

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues…

As the primary author of CSV.jl, I can help clarify on the posted issues: * #720: ended up not being an issue at all, but a misconfigured environment * #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug * #749, #734 are related to a new "beta" feature (CSV.Chunks) which…

I should have been a bit careful with this message, and mentioning issues that might be in beta. I don't want to specifically pick on csv.jl, but my experience of Julia has been every time I've worked on a significant program, we've hit an issue, either in core or a common library, certainly compared to Python (I avoid Javascript)

I agree that Julia is progressing quickly, but I think a lot of people (certainly myself) got burnt back in the earlier days of Julia when things were much less stable and changing rapidly and the language was (in many blogs) sold as ready for use.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#32

Post by karbarcca »

mcrad wrote: post #7

> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?

It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7.

I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy large files).

The performance work in the CSV.jl package (that the article is about) was very much geared towards these kinds of production scenarios.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#33

Post by krastanov »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

I love python and work mostly in it, and I have some issues with the current state of Julia, but I think you are underestimating how much the python science stack has changed recently in a very unstable manner. There is so much cutting edge science that simply can not be done with numpy/scipy. A very big part of this has been the acceptance of reverse design and automatic differentiation in many engineering fields, for which you simply have to use theano/jax/tensorflow/pytorch (I am not talking about machine learning here). Of these, I had to switch my simulations away from Theano when it was deprecated, I could not use pytorch because it does not support complex numbers, and I had to switch from Tensorflow 1 to Tensorflow 2, and I had to write by myself ridiculously basic things like ODE solvers.

The existence of ad-hoc solutions like numba or cython also do not speak well of the limitations of the language. Most specialized science packages (qutip for instance, for quantum optics), reimplement tons of solver/integration/optimization logic and require plenty of cython (or C) code. Scipy does not play nice with most of the automatic differentiation libraries.

In that context (a pretty universal context), Julia is looking pretty stable (relatively) and future proof in comparison. E.g. you can not really interoperate with scipy/tensorflow in a meaningful high-performance way. The value proposition of Julia is that this will be possible, today, and hopefully when future new paradigms appear.

Also, on a tangent, Julia already has ODE solvers and Probabilistic Programming packages lightlyears ahead of what Python has.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#34

Post by krastanov »

bonoboTP wrote: post #25
ddragon wrote: post #20

Earlier quoted context omitted.

By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more dras…

> By that logic people should have kept using Fortran instead of Python There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight th…

> The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue.

Numpy leaves a couple of orders of magnitude of performance on the table, especially if you use small arrays and a lot of intermediary computation. It is terrible in terms of memory allocation overhead. Cython or @tensorflow.function fix much of that, but then you are not really using python anymore.

> They often don't make use of that out of a sense of pride and not invented here.

This is a lazy and rather hurtful accusation.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#35

Post by jjoonathan »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

> Why can't people just settle on some tech

Because underlying assumptions change.

Python still has an enormous amount of baggage from its single-threaded and fully-interpreted past (and present, really -- but I'd expect fans to argue). Should we live with the kludgey, limited workarounds? Or switch to an ecosystem built on a more modern foundation? Only time will tell -- but I won't shed any tears for python if we decide on the second option.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#36

Post by willvarfar »

Don’t get me started on Java performance!

Java made things fast at the macro level by making threading possible. Remember trying to do any kind of portable multi core in C/C++ in 2000? It was no surprise JVM languages took over and became what we’ve built Big Data on (Hadoop, Hive, spark, etc)

But they leave so much on the table for micro performance! Take a CSV or JSON parser for example - likely spends all its time extracting string objects!

Now the Big Data community mostly doesn’t notice; for example, the Parquet code is about 5x slower than the Orc code, apparently, but we all adopt Parquet anyhow.

I’ve been going through my java codebases doing the basic stuff like “let’s write a simple fast json parser that doesn’t make a lot of garbage” and you can easily half the number of boxes you need etc. sometimes order of magnitude improvements by writing java code that is more like C.

But there’s this level you cannot reach. You look longingly at the raw mechanical sympathy C enabled.

Perhaps Rust is the future? I expect we’ll just keep using more cores than necessary though.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#38

Post by oefrha »

xyzzy_plugh wrote: post #30
fermienrico wrote: post #12

Earlier quoted context omitted.

With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are rea…

> Python's PEP / RFCs Is this process actually fundamentally better? There are so many PEPs that are up to the reader to interpret. There are so many PEPs that contradict each other (just read some of the early PEPs). The whole thing feels basically meaningless, and toothless at this point. I think Go's proposal process nails balancing formalities and discussion/bikeshedding. They seem to be very productive at making…

I can’t even tell if this is a bad faith attack or genuine cluelessness.

No, language features, standard library modules, infrastructure implementations, standard APIs, development and community processes, etc. etc. aren’t “up to the reader to interpret”, aren’t “basically meaningless”, aren’t “toothless”, especially not “at this point”. Informational PEPs are informational.

python-committers also aren’t a misaligned minority of the community. If you feel committers don’t serve your interest, step up and start contributing.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#39

Post by karbarcca »

CJefferson wrote: post #31
karbarcca wrote: post #26

Earlier quoted context omitted.

As the primary author of CSV.jl, I can help clarify on the posted issues: * #720: ended up not being an issue at all, but a misconfigured environment * #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug * #749, #734 are related to a new "beta" feature (CSV.Chunks) which…

I should have been a bit careful with this message, and mentioning issues that might be in beta. I don't want to specifically pick on csv.jl, but my experience of Julia has been every time I've worked on a significant program, we've hit an issue, either in core or a common library, certainly compared to Python (I avoid Javascript) I agree that Julia is progressing quickly, but I think a lot of people (certainly mysel…

No worries. The CSV.jl package has just been around for 5 years now, has some 700 issues opened/resolved, has built up a pretty extensive test suite, and is used in production by a number of companies, so I just wanted to hopefully clarify its level of maturity :D

I can definitely understand getting burnt in Julia early days (I was around back then as well), but since 1.0, the package ecosystem has matured quite a bit IMO; most of the really popular packages are very stable and work similarly to popular packages in other languages.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#40

Post by RobinL »

If you want to read csvs fast in Python, you should consider using the Apache Arrow[0] csv reader. Depending on your number of CPU cores it can be 10x-20x as fast as the native pandas reader. [1]

More broadly, because Arrow is cross platform it can give you similar performance in many languages. And once the dataframe is in memory, you can share it between languages with no need for serialisation and deserialisation.

[0] https://arrow.apache.org/ [1] https://youtu.be/fyj4FyH3XdU?t=1036

Post reply on HN