Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

51–60 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#51
There is a ton of conflation in this post, yesterdays post, and the comments.

* Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that "10x faster" number.

* Comparing concrete object decoding (Python csv module) to array decoding (PyArrow). Producing concrete objects representing every row (e.g. tuples of strings) will always be slower because of the pressure it puts on the heap and memory. Storing 10 million rows with 10 cols of 8-byte floats in an array might only generate 800 MB of memory bandwidth, the equivalent concrete list-of-floats PyObjects would come out at 6160 MB bandwidth and a ton of CPU burned in the allocator. There are use cases where either representation is preferred, and using e.g. PyArrow's parser then simply iterating the result as concrete objects, worst case the result will be slower than directly decoding to PyObject to begin with.

* Lumping floating point decoding (a problem with large performance-correctness tradeoffs) in with CSV parsing. It's hard to decode floats both quickly and precisely, it's also impractical to describe in the context of a comparison of CSV parsers which language/implementation might decode floats better and why that is better.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#52
post #20

Earlier quoted context omitted.

By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more dras…

> By that logic people should have kept using Fortran instead of Python There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight th…

I feel like you considering Python as having few real pain points in data science as either lack of knowledge of other languages or imagination/ambition. I do work on Python for data science/engineering in a production environment and I do find many of those all the time. Python does not feel like pouring my thoughts because I cannot write Python directly without taking hours to handle a few million points of data, so I have to juggle with multiple dialects (pandas, numpy, pytorch, tensorflow), and Python ends up being one of the languages that I need some documentation at all times. And then that documentation is also not obvious what's the input, do I need to give a tuple, or perhaps a dict or even a list because typing is very recent so the ecosystem isn't up to date and sometimes I can't even type my code properly because I legitimately don't know what a library function returns. And even with mypy I keep finding errors that I can only find after deploying (I don't blame Python on this one, and Julia isn't really better, but I can still dream of something that is dynamic when I want but still safe). Also Python for a dynamic language has pretty mediocre interactive story (using ptpython since the default repl is unusable), even simple things like copy pasting to a REPL can end up being a pain because of indenting, and the repl is far away from Lisp, Clojure and even Julia and Elixir. And Elixir also makes me especially disappointed with Python's multithreading story (in Elixir it's so natural that it really is the "pouring your throughts on the screen" for distributed).

It ended up being a rant, but I have many pain points with Julia as well, but it's still a new language that has more space to evolve and find ways to solve them, and if the solution is yet another language that solves all of them, I'll quickly jump. I spend a lot of time programming, so any significant improvements on the usage of my time is worth the effort in learning.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#53
post #40

If you want to read csvs fast in Python, you should consider using the Apache Arrow[0] csv reader. Depending on your number of CPU cores it can be 10x-20x as fast as the native pandas reader. [1] More broadly, because Arrow is cross platform it can give you similar performance in many languages. And once the dataframe is in memory, you can share it between languages with no need for serialisation and deserialisation.…

I agree; if I needed to parse CSVs in python and could utilize the arrow format, I would definitely use pyarrow.

I actually recently finished support for reading/writing the arrow format in Julia (https://github.com/JuliaData/Arrow.jl), and it's automatically integrated with the CSV.jl package; so you can do `Arrow.write("data.arrow", CSV.File("data.csv"))` and convert a csv file to arrow format directly. I'm very bullish on arrow as a standard binary data format for the future.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#54
post #21
post #16

Well, Julia is compiled .Python and R are both interpreted. I think that simdjson is fastest json parser out there( Parsing gigabytes of JSON per second ). https://github.com/simdjson/simdjson So,if I use Python binding for simdjson , then parsing of json in python must be very much faster than the fastest one mentioned in the above post. So,at the end,it depends on implementation not only on language.

Python's CSV parser is in C. I don't think a native python CSV parser would be nearly as fast.

You can write slow programs in all languages. There are many cases where Julia's language features enable higher performance than in comparable C libraries.

Have a look at Steve Johnson's keynote from JuliaCon 2019: https://www.youtube.com/watch?v=mSgXWpvQEHE

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#55
post #44

Here is the thing made me a happier person this year: I realized when CSVs take too long to load, I should not be using them in the first place! I love SQLite now <3

It's always a transcendental experience to move data from CSV into any more structured/flexible format!

And fast! I got easily 3 orders of magnitude speedup. I used to load CSVs with pandas and do typical filtering operations... oh boy, some of the stuff would take a hole morning, not to mention the memory usage. I do all that in minutes now.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#56

Earlier quoted context omitted.

With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are rea…

One advantage I've noticed at least (having been a contributor to the Julia language itself), is that it at least makes contributing to the language very approachable. Granted you see lower quality proposals from time to time, but in general, I even question whether I myself would have gotten involved or been brave enough to propose language features if I would have had to do such a rigorous, formal PEP writeup. In s…

> if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not.

If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. Not sure about your definition of “more flexible”, if that entails breaking existing code then it may rightfully meet resistance.

Not sure why you’re implying other languages’ proposal processes are about pleasing a single core developer or playing some politics game. Sounds like either you have a lot more experience than I do and hence have been exposed to politics games or anything else non-meritocratic (that I haven’t sensed in CPython development, at least), or you just don’t have as much experience contributing to other languages.

(I'm a bit cranky today. Sorry if I sound aggressive.)

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#57

Don’t get me started on Java performance! Java made things fast at the macro level by making threading possible. Remember trying to do any kind of portable multi core in C/C++ in 2000? It was no surprise JVM languages took over and became what we’ve built Big Data on (Hadoop, Hive, spark, etc) But they leave so much on the table for micro performance! Take a CSV or JSON parser for example - likely spends all its time…

Take a deep look at Julia! Adding multithreaded parsing capabilities to CSV.jl was really a joy; basically just chunking up the file and spawning threaded tasks to process each chunk with the existing parsing code. My favorite favorite thing about developing in Julia is the ability to write "high-level" type code and usually get decent performance, BUT THEN have the ability to fine tune for that extra boost of perfor…

(Some of the comments in https://news.ycombinator.com/item?id=24736559 talk about new lines inside fields, making multi-threading difficult)

(My quick search to find that recent story on HN made me find this too https://news.ycombinator.com/item?id=24740527)

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#58
post #20

Earlier quoted context omitted.

By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more dras…

> By that logic people should have kept using Fortran instead of Python There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight th…

> The only issue with Python is speed

That's a big one. Memory usage and typing are other issues. That it doesn't really run in the browser is another. Or that it has more issues than C++ when trying to write a portable GUI.

It's fine for education and prototyping.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#59

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues…

With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are rea…

On the other hand, many important things move at a grinding pace in Python. Packaging and performance are two major issues for just about everyone I’ve spoken with who uses Python in a serious capacity, and these have been notorious problems for decades with no end in sight. Most of this seems to boil down to an unwillingness to encourage the ecosystem towards a narrower packaging format (“sorry, downloading a package and running setup.py just to determine the dependency set is insane and we’re going to deprecate that going forward”) or a narrower C extension interface that would be compatible with more aggressive optimizations. There’s value in stability, but if the governance model can’t pass these kinds of slow, steady changes for paramount issues, then perhaps new languages shouldn’t look to Python as a model for a well-governed language.

I would suggest Rust—I’m not a fanboy by any means, but Rust is a large, rapidly growing project that is unparalleled with respect to its rate of improvement as far as I’m concerned. I know recommending Rust for things is a cliche (esp “rewrite it in Rust”), but I would defy anyone to find a better governed language.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#60

Earlier quoted context omitted.

As the primary author of CSV.jl, I can help clarify on the posted issues: * #720: ended up not being an issue at all, but a misconfigured environment * #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug * #749, #734 are related to a new "beta" feature (CSV.Chunks) which…

I should have been a bit careful with this message, and mentioning issues that might be in beta. I don't want to specifically pick on csv.jl, but my experience of Julia has been every time I've worked on a significant program, we've hit an issue, either in core or a common library, certainly compared to Python (I avoid Javascript) I agree that Julia is progressing quickly, but I think a lot of people (certainly mysel…

Same here. Burned quite a few times trying out Julia stuff.

Last time it was because cold start times were unbearable (compared to Python/R/Stata/Matlab). My goal was to compare how regressions in a few software packages behaved with difficult datasets, so I had to open each of those, run a snippet of code, and log the output. Here, Julia's cold start (and importing the CSV, GLM, etc libraries) took way longer than all of the other tools together.

Post reply on HN