Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

Post reply on HN
Print view View original

61–70 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#61

Post by karbarcca »

mrfusion wrote: post #9

I’d really think all these languages would have the bottleneck of disk read speed. So how can Julia be that much faster? Are the others really that inefficient?

The biggest difference in these benchmarks comes down to how multiple threads are leveraged; (disclaimer: primary CSV.jl author here).

In CSV.jl, it was relatively straightforward to add multithreaded parsing support; we chunk up the file, find row starts, then spawn a threaded task to process each chunk. In data.table (fread), they have a constraint of R having a global String intern store; so it fundamentally restricts the multithreading capabilities by requiring strings to be interned sequentially. In pandas, there are similar nontrivialities jumping between the C++ source and python object world that adds a lot of complexity (and probably explains why no one has made the effort to do multithreaded parsing). It's interesting to note, however, that the apache arrow project (pyarrow package in Python), has a multithreaded csv parser integrated with the project. It provides similar performance to CSV.jl because it was built from the ground up to process chunks on multiple threads into "arrow tables". Unfortunately, it's tied very specifically to the arrow project, so pandas doesn't benefit from its work! It's one of my favorite features about Julia that CSV.jl automatically integrates with other "data table formats"; i.e. ODBC.jl, SQLite.jl, MySQL.jl, Arrow.jl, DataFrames.jl, etc. They can all leverage CSV.jl as "their csv parser" because it's seamlessly integrated via well-established "table" apis.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#62

Post by genomez4fun »

varbhat wrote: post #16

Well, Julia is compiled .Python and R are both interpreted. I think that simdjson is fastest json parser out there( Parsing gigabytes of JSON per second ). https://github.com/simdjson/simdjson So,if I use Python binding for simdjson , then parsing of json in python must be very much faster than the fastest one mentioned in the above post. So,at the end,it depends on implementation not only on language.

julia is interpreted -- it's type system enables the speedups often presented in benchmarks.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#63

Post by echelon »

I think there's a strong chance that Swift or Rust take a lot of Python's data science cake. Both of them are extremely fast, concrete, and have lots of investment being poured into numerics and fitting into the Python/ML ecosystem.

I don't think Julia or R are going to steal this away. In fact, I think Julia is a major turn off to engineers with some of the bizarre choices they made (eg. 1-based indexing to appease math-centric backgrounds). If you don't excite engineers, you're not going to get as many library and optimization contributions.

Julia feels like it came too late in the game and the newer entrants are making quick strides in the mathematics space.

I could be totally wrong. This is just my take, and I have no stake in this fight other than I want a good, safe, highly-compatible, and fast ecosystem to work with.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#64

Post by karbarcca »

andi999 wrote: post #41
karbarcca wrote: post #32

Earlier quoted context omitted.

It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7. I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy la…

I am curious: how do you transfer these amounts of data fast?

Unfortunately almost exclusively via http rest apis. It's not great, but it's the lowest common denominator between the vast "ingestion" service we've built (connectors to web apis, local application for local file upload, raw api endpoints, etc.).

We've started exploring the apache arrow format as a compressible binary format with a dedicated wire format just to cut down on parsing processing costs.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#65

Post by spockz »

dmw_ng wrote: post #51

There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that…

So what about using Flatbuffers[1] to avoid memory allocation of the data and only have the structure allocated additionally?

[1]: https://google.github.io/flatbuffers/

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#66

Post by ku-man »

I'm glad Julia does exist. It's great to have a free and open-source programming language focused on scientific computing.

What I dislike from the Julia community is the unhealthy fixation they have with Python and R. Virtually, no comment or blog-post is written without mentioning how slow, inefficient, inappropriate, inelegant is Python or R. Somehow, the Julia community convinced itself that best way to attract more users is to constantly put down Python and R. A sad state of affairs definetely.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#67

Post by karbarcca »

oefrha wrote: post #56
karbarcca wrote: post #45

Earlier quoted context omitted.

One advantage I've noticed at least (having been a contributor to the Julia language itself), is that it at least makes contributing to the language very approachable. Granted you see lower quality proposals from time to time, but in general, I even question whether I myself would have gotten involved or been brave enough to propose language features if I would have had to do such a rigorous, formal PEP writeup. In s…

> if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. Not sure about your definition of…

No worries, it doesn't come off that aggressive ;)

Yes, I've definitely been involved in other projects where a single core developer can do a little too much "imposing" of their will. i.e. if they happen to have a personal preference one way or another on things, they not accept or even entertain certain changes.

I wouldn't expect most large, mature projects to be this way (otherwise they probably wouldn't have grown as large or attracted enough of a community!), but it's certainly a problem with some projects.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#68

Post by throwaway894345 »

bonoboTP wrote: post #14

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperab…

> Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare.

This is a sunk cost argument. Python’s stack got us a long way, but it still falls short in a lot of ways. First of all, it is eager so you can’t build up an entire query, have the system optimize it, and then execute it for greater performance. Secondly, you can’t easily parallelize operations on the same dataframe. Thirdly, it’s poorly suited to running in a web process. Fourthly, it’s all written in C/C++, so debugging, building/packaging/installation (esp on niche systems), etc is a pain and it prevents many of the optimizations Julia and others can do natively, etc. Fifthly, the APIs were horribly designed (especially matplotlib, good grief). Etc.

I don’t know about Julia in general, but in most domains there’s a middle ground where people can have nice, thoughtfully designed tools (albeit with minor issues), and endless churn. It certainly feels like the scientific toolchain is one of those areas—fix a few of these enormous, glaring problems and then enjoy some stability.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#69

Post by elcomet »

genomez4fun wrote: post #62
varbhat wrote: post #16

Well, Julia is compiled .Python and R are both interpreted. I think that simdjson is fastest json parser out there( Parsing gigabytes of JSON per second ). https://github.com/simdjson/simdjson So,if I use Python binding for simdjson , then parsing of json in python must be very much faster than the fastest one mentioned in the above post. So,at the end,it depends on implementation not only on language.

julia is interpreted -- it's type system enables the speedups often presented in benchmarks.

It's a bit more complex. Both python and julia are compiled to bytecode.

Then python bytecode gets interpreted, but I think that julia bytecode is actually compiled with a JIT compiler.

So julia is not really interpreted.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#70

Post by oefrha »

karbarcca wrote: post #67
oefrha wrote: post #56

Earlier quoted context omitted.

> if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. Not sure about your definition of…

No worries, it doesn't come off that aggressive ;) Yes, I've definitely been involved in other projects where a single core developer can do a little too much "imposing" of their will. i.e. if they happen to have a personal preference one way or another on things, they not accept or even entertain certain changes. I wouldn't expect most large, mature projects to be this way (otherwise they probably wouldn't have grow…

> it's certainly a problem with some projects.

Projects, of course. Ages ago I quit a notable project partly because of maintainer team politics, so I'm no stranger. But I thought we were specifically talking about language improvement processes of non-niche programming languages (a specific one, even).

Post reply on HN