Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

Post reply on HN
Print view View original

21–30 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#21

Post by bjoli »

varbhat wrote: post #16

Well, Julia is compiled .Python and R are both interpreted. I think that simdjson is fastest json parser out there( Parsing gigabytes of JSON per second ). https://github.com/simdjson/simdjson So,if I use Python binding for simdjson , then parsing of json in python must be very much faster than the fastest one mentioned in the above post. So,at the end,it depends on implementation not only on language.

Python's CSV parser is in C. I don't think a native python CSV parser would be nearly as fast.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#22

Post by vikramkr »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

Because people want to? I think a big part of it is that people like trying new things and trying to differentiate in some way. In something as competitive as web dev, it makes sense that the churn in frameworks is so high, because everyone is looking for just that little edge or unique special sauce that makes development a bit easier or makes the competitor a bit more obsolete, or attracts talent thats just a little bit better. For someone out there, a little edge in CSV parsing is going to make their lives just a bit easier and help them do just a bit more than the competing lab, and grad students/labor is cheap, so why not have a grad student play around with Julia and see if you can get anything out of it?

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#23

Post by jdietrich »

mynegation wrote: post #18
mrfusion wrote: post #9

I’d really think all these languages would have the bottleneck of disk read speed. So how can Julia be that much faster? Are the others really that inefficient?

Because it is not a bottleneck. Modern SSDs can read data at more than 500 megabytes per second, which, on a single thread gives you just about 6-8 clock cycles to process a single byte. That’s not a lot of CPU instructions so it will take quite a number of threads to handle this bandwidth.

The SSD in my current workstation tops out at 5GB/s sequential reads and 680kIOPS. That's a modestly high-end consumer M.2 drive, not enterprise exotica. SSDs are really, really fast.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#25

Post by bonoboTP »

ddragon wrote: post #20
bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more dras…

> By that logic people should have kept using Fortran instead of Python

There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight the language. The syntax is very clear, concise, "Pythonic" is a word of praise. As I said, this kind of debate is rarely productive and devolves to religious arguments, but that's my stance.

The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue.

> and can start with increased knowledge from previous attempts

They often don't make use of that out of a sense of pride and not invented here.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#26

Post by karbarcca »

CJefferson wrote: post #6

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues…

As the primary author of CSV.jl, I can help clarify on the posted issues:

* #720: ended up not being an issue at all, but a misconfigured environment * #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug * #749, #734 are related to a new "beta" feature (CSV.Chunks) which allows iterating over large CSV files in batches. I've been trying to track down the issues 2 people have reported, but haven't been able to reproduce on the same large file. Once we iron out some of those issues, we'll mark the feature as "out of beta".

I agree that Julia packages in general are still evolving and you might run into issues, but at the same time, I strongly believe it's reached a similar maturity in most ways with other language package ecosystems. For example, I use a lot of Javascript/Python frameworks/packages and at least for me, I tend to run into corner case bugs/issues as often as I do for the most common/mature Julia packages.

Compared with, say, data.table in R, or pandas in python, one of the things I enjoy most about Julia packages is that they're almost exclusively written in pure Julia. Having had to dive into data.table/pandas source C/C++ + language binding glue code is a huge pain when trying to track down bugs, so I feel like my knowledge of Julia "goes further" in that if I run into a package bug, it's relatively much easier to track down what's going on and even submit a pull request to fix!

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#29

Post by crispyambulance »

qaq wrote: post #2

Julia is a hidden gem Once ecosystem for web dev matures Julia will be the killer lang for building web apps

  > Once ecosystem for web dev matures Julia will be the killer...
Looking back at history, javascript was put together in haste, almost like an afterthought. Even the name was a stupid marketing tack-on.

Yet it stuck and kept going even when people became sick of it. It has reinvented itself several times over, and in spite of the emergence of multiple superior technologies that were specifically targeted at web-development, it still stands at the top of hill.

I guess it would be properly ironic that something designed for mathematics would take down javascript for web-dev, but I doubt it. It would be amusing though.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#30

Post by xyzzy_plugh »

fermienrico wrote: post #12
CJefferson wrote: post #6

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues…

With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are rea…

> Python's PEP / RFCs

Is this process actually fundamentally better? There are so many PEPs that are up to the reader to interpret. There are so many PEPs that contradict each other (just read some of the early PEPs). The whole thing feels basically meaningless, and toothless at this point.

I think Go's proposal process nails balancing formalities and discussion/bikeshedding. They seem to be very productive at making the language better, thinking critically and avoiding regrets, rather than pandering to the very vocal and often misaligned minority of the community.

Post reply on HN