Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

Post reply on HN
Print view View original

11–20 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#11

Post by gameswithgo »

mcrad wrote: post #7

> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?

I can think of many cases where that is useful. It is important to remember not all workflows and needs are the same as yours.

I don’t really understand tour web dev comment either. Web devs are notoriously not interested in performance.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#12

Post by fermienrico »

CJefferson wrote: post #6

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues…

With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are really nice but IMO it is missing structure, discipline, hardline regimented approach of confronting difficult problems, etc. One feels like hearing a supreme court case and the other a street brawl to decide upon important issues.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#13

Post by fermienrico »

gameswithgo wrote: post #11
mcrad wrote: post #7

> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?

I can think of many cases where that is useful. It is important to remember not all workflows and needs are the same as yours. I don’t really understand tour web dev comment either. Web devs are notoriously not interested in performance.

Then it goes both ways right? I think we need to also say "For most people Python is fine, if you need that extra speed, try Julia" instead of blanket praises of Julia as true competitor to Python (It doesn't even come close IMO).

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#14

Post by bonoboTP »

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades.

Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare.

I know there's some interoperability between Julia and Python, but a lot of stuff is not interoperable, and there are annoying differences in array indexing etc.

Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch?

This causes a lot of artificial bit rot, there's a never ending treadmill of just porting well-developed concepts and mature code to slightly changed representations. It's the busywork of tech.

Does the scientific community also want to follow web dev in their insane churn of frameworks and libraries where last week's best practice is laughable ancient stone-age primitiveness today?

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#16

Post by varbhat »

Well, Julia is compiled .Python and R are both interpreted.

I think that simdjson is fastest json parser out there( Parsing gigabytes of JSON per second ).

https://github.com/simdjson/simdjson

So,if I use Python binding for simdjson , then parsing of json in python must be very much faster than the fastest one mentioned in the above post.

So,at the end,it depends on implementation not only on language.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#17

Post by dagw »

mrfusion wrote: post #9

I’d really think all these languages would have the bottleneck of disk read speed. So how can Julia be that much faster? Are the others really that inefficient?

Without having looked at it, it might also be a case that the other libraries spend more time handling more obscure corner cases and doing more robust error handling.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#18

Post by mynegation »

mrfusion wrote: post #9

I’d really think all these languages would have the bottleneck of disk read speed. So how can Julia be that much faster? Are the others really that inefficient?

Because it is not a bottleneck. Modern SSDs can read data at more than 500 megabytes per second, which, on a single thread gives you just about 6-8 clock cycles to process a single byte. That’s not a lot of CPU instructions so it will take quite a number of threads to handle this bandwidth.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#19

Post by Datenstrom »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

There is an immense value in most of the ecosystem converging on Python over the last decade and especially with leaving MATLAB behind in the dust. I don't think decade long transitions from old tech to new tech is a bad time scale or even close to that of the churn of web technologies. Julia offers enough of an improvement over Python to warrant a switch over the next 5-10 years and leave Python behind in the same way. I believe Julia offers the same level of improvement over Python as Python does over MATLAB.

I also believe that it has drawn too much from MATLAB only for the sake of being familiar to that group and not the computer science and software engineering communities though such as the one based indexing you mentioned. Unfortunately that ship has already sailed and we will likely have to wait another 10 years for the next one to hopefully fix those problems.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#20

Post by ddragon »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more drastic Python 2 -> 3, or adding things indefinitely until you get effectively many languages/dialects in one like C++).

New languages on the other side don't have as much baggage (and can start with increased knowledge from previous attempts) so they are able to freely potentially target better local maxima that old languages cannot anymore. Plus in this particular case, Python was not really designed to target the scientific domain (nor ML which did not even exist in the way we use nowadays), it was repeatedly retrofitted with possibly a lot more effort than would be necessary to create something better from scratch.

Post reply on HN