I'm still looking for a nice way to read password protected xlsx files in R....
CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
81–90 of 236 posts
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#82The vroom R package is likely the fastest R package in read speed, not the ones used in this article. vroom was 13.1x faster than R's fread/data.table based on the read performance benchmark in https://cran.r-project.org/web/packages/vroom/vignettes/benc... so may be similar or faster in read speed vs julia.
vroom doesn't really read files initially though. Which is why the comparison isn't 1:1.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#83Earlier quoted context omitted.
I should have been a bit careful with this message, and mentioning issues that might be in beta. I don't want to specifically pick on csv.jl, but my experience of Julia has been every time I've worked on a significant program, we've hit an issue, either in core or a common library, certainly compared to Python (I avoid Javascript) I agree that Julia is progressing quickly, but I think a lot of people (certainly mysel…
Wouldn't this statement be true for any new technology? I don't doubt what you say about being burnt when adopting in the early days. However, it is simultaneously true that people have successfully used Julia in several commercial applications and significant research codebases for many years now. I routinely point people to the Julia Computing case studies for this reason: https://juliacomputing.com/case-studies/ W…
I've generally found the Julia community less welcoming than some (Rust, Python), although much better than the "classics" (C,C++,Java and friends).
For example, picking the first question I can see on discourse (which isn't a clear technical question, it was number 3) is "What is the status of debugger?". The first answer is "I've used Julia for 3 years and I don't need a debugger" whereas searching for similar questions on the Rust discourse tends to find answers.
This isn't a complete investigation, it might just be me generalising from my small number of interactions with the Julia and Rust communities.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#84I'm pretty sure you can add multiprocessing parsing of csv with Python and Pandas. I think I've done it before.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#85If your CSV file is so large parsing it takes a nontrivial amount of time, why didn't you use a binary format?
And if someone else uses CSV? The people processing data are almost never the people who generated the data.
Probably still worth it to convert to a binary format, assuming it's read more than once or twice and is bigger than 10 MB or so.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#86Earlier quoted context omitted.
One advantage I've noticed at least (having been a contributor to the Julia language itself), is that it at least makes contributing to the language very approachable. Granted you see lower quality proposals from time to time, but in general, I even question whether I myself would have gotten involved or been brave enough to propose language features if I would have had to do such a rigorous, formal PEP writeup. In s…
> if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. Not sure about your definition of…
> If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not.
I wouldn't be so sure. Search for `site:bugs.python.org "maintenance overhead"` on your favourite search engine.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#87> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?
It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7. I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy la…
Right - sounds like you have more of a production support role vs. a data analysis workflow kind of task. Tacking on "exploratory" is helpful but I'm still concerned that you misuse the overall concept of analysis. It's decision-making task, which is practically the opposite of production support.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#88Wonder how it performs compared to the Jd csv reader
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#89Earlier quoted context omitted.
> if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. Not sure about your definition of…
> If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. I wouldn't be so sure. Search for `site:bugs.python.org "maintenance overhead"` on your favourite search engine.
8 results total, with exactly one entry that’s remotely relevant (i.e. performance related, patch provided) — bpo-21955. You could have just linked that. And this happens to be a patch from a core dev. And it’s not comparable to submitting a performant implementation of a PSL module like csv. You have chosen a pretty convincing example, I don’t know what I’m supposed to be convinced of though.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#90Earlier quoted context omitted.
We need more decades-stable stuff similar to LAPACK and BLAS, but for one step higher-level things. Just imagine if people had to rewrite all that numerical computation for every new language. Leaving some performance critical stuff in C is okay. We can expose them to a variety of languages then. > away from Theano [...] I had to switch from Tensorflow 1 to Tensorflow 2 I feel you. But I think this has more to do wit…
One of the neighboring comments mentioned something about changing assumptions. This is what limits the creation of decades-stable stuff. If you leave the performance crucial stuff in C, that stuff will not operate with the novel code you wrote (autodiff, probabilistic programming, GPU/TPU hardware, etc). Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actua…
The "using Julia from other languages" story doesn't look that great though: https://docs.julialang.org/en/v1/manual/embedding/
Mostly the bits about how you have to workaround the Julia GC.
So it doesn't seem like a great option for libraries that might have lots of different bindings, where C/C++/Fortran (and maybe Rust?) seem to be the only reasonable choices.