Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

Post reply on HN
Print view View original

71–80 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#71

Post by ViralBShah »

dmos62 wrote: post #5
qaq wrote: post #2

Julia is a hidden gem Once ecosystem for web dev matures Julia will be the killer lang for building web apps

> Julia will be the killer lang for building web apps That would be fun, but Julia's community aren't web devs. Julia spawned around very specific needs of scientific computing, which is characterised by a short-running daemon or a script-type interpreter. A web server is a long-running process. Not knowledgeable enough about Julia to tell how it lends itself to server uses, but heard hearsay that it's problematic.

See Jacob Quinn's (karbarcca) other package: https://github.com/JuliaWeb/HTTP.jl

It is quite common for scientists and engineers to want to present the findings of their work or share them with others who are non-coders. Thus you get tools like Jupyter, Shiny, Pluto, etc.

There are several long running Julia processes in real applications. While there's no doubt that there is work to be done, the set of these applications is not the null set either.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#72

Post by qaq »

akerro wrote: post #15
qaq wrote: post #2

Julia is a hidden gem Once ecosystem for web dev matures Julia will be the killer lang for building web apps

Rust is already killing old web app building technologies!

Julia is a good bit easier to pick up than Rust. I like Rust btw. but say at my job chances of us adopting Rust would be close to 0. Julia is really easy to pick up so would be much easier sell once ecosystem is mature.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#73

Post by secondcoming »

spockz wrote: post #65
dmw_ng wrote: post #51

There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that…

So what about using Flatbuffers[1] to avoid memory allocation of the data and only have the structure allocated additionally? [1]: https://google.github.io/flatbuffers/

Binary will always beat the pants off human-readable formats. I doubt Excel can export to Flatbuffer, Protobuf, MessagePack, Parquet, etc. So CSV is pretty much here to stay, even though correctly parsing it is more difficult that it seems.

To gain even from parallelism, make all your rows a fixed size. Arbitrary length fields aren't your friend if you want max speed.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#74

Post by nemothekid »

bonoboTP wrote: post #25
ddragon wrote: post #20

Earlier quoted context omitted.

By that logic people should have kept using Fortran instead of Python (which was the new guy a couple decades ago). But Fortran will never be Python, and Python will never be Fortran, no matter how much maturity/improvements. Because maturity means getting close to it's potential (local maxima), without the ability to drastically move to a hypothetical global maxima (without breaking all it's legacy, like a more dras…

> By that logic people should have kept using Fortran instead of Python There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight th…

>They often don't make use of that out of a sense of pride and not invented here.

You are handwaving away a big reason why there is research into non-Python solutions. Unless you work at FAANG, or some other institution where developer time is much, much greater than compute time; simply using a more performant solution can save either in days of waiting or thousands of dollars a month in compute.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#75

Post by leovailati »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

Disruption is the price of progress. Like cars with electric motors vs combustion engines, sometimes you have to start from scratch. You can't just keep improving combustion engines forever, you eventually reach a technological limit.

To your point though, I do see a lot of unnecessary disruption particularly in the web dev world. I think people like working on new stuff. It's exciting to take the first steps, see order of magnitude improvements every commit.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#76

Post by ssivark »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

> Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch?

At what level of language improvement does it warrant rebuilding things? Eg: What if there has actually been a change as revolutionary as multiple dispatch based on a good type system, which cannot be backported to Python, and provides huge advantages for a diverse set of applications? For many users, it would be crazy not to switch... especially given that it’s actually much easier to write performance and flexible libraries in the Julia ecosystem (writing Julia code) than in the Python ecosystem (writing C/C++/Cython code).

You might be succumbing to a bit of the blub paradox here... Python might be good enough for all your needs, but people are justifiably enamored with Julia because it brings some really exciting capabilities to the table which allows them to solve serious impediments they experience, not just because it is a shiny new thing.

Rhetorical question: Now that we have all this infrastructure for gasoline based cars, is that reason enough to hold back the “churn” to electric cars?

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#77

Post by burntsushi »

I posted this comment on lobste.rs when this came up:[1]

It kind of looks like Julia’s CSV parser is cheating: https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa5...

It’s doing parallel parsing, but I’m pretty sure their technique won’t work for all inputs. Namely, they try to hop around the CSV data and chunk it up, and then parse each chunk in a separate thread AIUI. But you can’t do this in general because of quoting. If you read the code around where I linked, you can see they try to be a bit speculative and avoid common failures (“now we read the next 5 rows and see if we get the right # of columns”), but that isn’t going to be universally correct.

It might be a fair trade off to make, since CSV data that fails there is probably quite rare. But either I’m misunderstanding their optimization or they aren’t being transparent about it. I don’t see this downside anywhere in the README or the benchmark article.

[1] - https://lobste.rs/s/zksa0f/loading_csv_file_at_speed_limit_n...

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#78

Post by mcrad »

gameswithgo wrote: post #11
mcrad wrote: post #7

> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?

I can think of many cases where that is useful. It is important to remember not all workflows and needs are the same as yours. I don’t really understand tour web dev comment either. Web devs are notoriously not interested in performance.

Maybe I don't understand web dev; only meant that when you have many users besides yourself (analyzing data) then performance matters more.

It's important to remember, wtf?

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#79

Post by ggrothendieck »

The vroom R package is likely the fastest R package in read speed, not the ones used in this article. vroom was 13.1x faster than R's fread/data.table based on the read performance benchmark in https://cran.r-project.org/web/packages/vroom/vignettes/benc... so may be similar or faster in read speed vs julia.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#80

Post by liuliu »

dmw_ng wrote: post #51

There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that…

For 1), it is simple, but not trivial to parallelize csv parsing correctly, hence the yesterday’s post.

Otherwise, agree on 2 and 3. Object wrapping, number decoding can be slow on the whole “file to dataframe object” path, but trivially parallelizable.

Post reply on HN