From where?
I couldn’t see anywhere in the article that specifies where the data is stored.
CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
171–180 of 236 posts
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#172Earlier quoted context omitted.
> 2. I don't entirely follow this point. Perhaps using PyArrow's parser would be faster than what is timed here, but is that what the typical Python data science user would do? I am a Python data science user. If data gets big enough such that loading time is a bottleneck, I use parquet files instead of CSV, and PyArrow to load them into pandas. It’s a one line change. The creator of Pandas is now leading the Arrow p…
Perhaps not directly relevant to your point here, but thought it would be interesting to anyone following along. Jacob Quinn (karbacca) also has a Julia package for integrating Julia into the Arrow ecosystem: https://github.com/JuliaData/Arrow.jl
Thanks Viral. To be clear, I’m a python user who’s cheering for Julia, because I live the problems of python and do see the potential of Julia as a better path. But unfortunately I’m not prepared to be the early adopter (at least in my day job), and will wait until other, braver users have sanded off the rough edges. God speed and good luck.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#173Earlier quoted context omitted.
> Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. This is a sunk cost argument. Python’s stack got us a long way, but it still falls short in a lot of ways. First of all, it is eager so you can’t build up an entire query, have the system optimize it, and then execute it f…
> Fifthly, the APIs were horribly designed (especially matplotlib, good grief) I've seen comments like this fairly often, but am not really sure what about matplotlib's API makes it so bad. Hoping to learn why. The only equal-basis competitor to matplotlib that I'm familiar with is Matlab, which is definitely worse. Declarative plotting libraries like ggplot2 have a nicer API, but the grammar of graphics approach is…
The API is state full and the args it accepts are arbitrary. It just tries to guess what you want it to do based on the types of the args you pass and their order. I’m not sure which plotting API is better, but there’s nothing unique about plotting in all of computing that requires an insane API.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#174Earlier quoted context omitted.
On the other hand, many important things move at a grinding pace in Python. Packaging and performance are two major issues for just about everyone I’ve spoken with who uses Python in a serious capacity, and these have been notorious problems for decades with no end in sight. Most of this seems to boil down to an unwillingness to encourage the ecosystem towards a narrower packaging format (“sorry, downloading a packag…
I find Rust really difficult to get into :( Python, you're right has a bunch of problems with datetime and packages, etc. but when it comes to productivity, it is insane how amazing Python is. Also being able to hack internals of python library and "batteries included" philosophy is why it is the most popular language in the world now overtaking Java.
Rust does have a steep learning curve and Python is a very productive language... until you are the one responsible for builds and deploys, or performance, or concurrency, or quality, or etc. I think you’ll find that Go hits a nice sweet spit between all of those competing concerns even if it isn’t the best at any of them (in general; however, probably not for data science in particular).
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#175> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?
It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7. I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy la…
> process terabytes of csv data every day
All stored on NVMe SSDs? Because unless you have really fast IO the CSV parser isn’t going to be the bottleneck.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#176Earlier quoted context omitted.
Yes, Gadfly does have a large compile time. Reducing the time to first plot (essentially compilation time) has been a major focus for the 1.5 release. People often use Plots.jl or PyPlot which have significantly lesser compile times.
@ViralBShah and @StefanKarpinski I've had similar problems and so have many of my colleagues in silicon valley. No matter how you spin it and whatever benchmarks you show - Julia is a very slow fast language. :) The UX of Julia needs major work - everything is just slow . If you've used Python for 10 years and it is like running through molasses. Benchmarks are meaningless for the most part. This is what bothers me a…
While I hope they'll keep the pace of improving the JIT lag (perhaps even adding a fast optional interpreter or AoT compilation of parts of code), the benchmarks are fine considering compilation is a one time paid ticket, while runtime is paid by usage. It's not just high compute, if I set up a website in Julia on a docker after the "cold start" (which can be handled before using packagecompiler if I don't want any client to feel the slow JIT compilation, as I obviously should) then it should run at the benchmarked runtime speed even if each request is very short (as the compilation price was paid already). That applies to most services in general, and even during development Julia programmers eventually learn not to cold start all the times (keeping the session open until you stop programming, so you again only pay the cost once).
Of course, any short task that must cold start like a bash script isn't convenient at all (and packagecompiler will generate a fat binary that runs fast, but it's still inconveniently fat if it's for something simple). And Julia's first impression is really important, which the "time to first plot" seems to affect the most (as you need to go deeper on the language to adapt to it's unusual workflow), so I agree that it's a UX priority.
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#177Earlier quoted context omitted.
Yes, Gadfly does have a large compile time. Reducing the time to first plot (essentially compilation time) has been a major focus for the 1.5 release. People often use Plots.jl or PyPlot which have significantly lesser compile times.
time to precompile Plots: 138s julia> @time Plots.scatter(data.eq_site_limit, data.hu_site_limit) 5.461595 seconds subsequent calls report significant speed ups compared to matplotlib, but render about as quickly julia> @time Plots.histogram(data.eq_site_limit) 2.143983 seconds
There are 3 different phases:
1. Precompilation time: Only once when you install or upgrade a package.
2. First running time (that includes compilation time): When you call a function like `plot` the first time in a session
3. Second and subsequent running times: The compiled code is now in the cache, and what you see is the actual running time.
Thus what you are reporting here as precompilation (1) is not something that a user will see every time. What they will see is (2) for the first time in a session and (3) on subsequent plots. Could you confirm this is what you see - to ensure that those following along understand the terminology?
Here are the usage times I see when I use Plots on a daily basis in a Julia session:
~ exec '/Users/viral/Desktop/Julia_Releases/Julia-1.5.app/Contents/Resources/julia/bin/julia'
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.2 (2020-09-23)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release julia> @time using Plots
11.341632 seconds (20.21 M allocations: 1.108 GiB, 2.87% gc time)
julia> @time plot(1:10,1:10)
2.291147 seconds (3.24 M allocations: 165.569 MiB, 7.80% gc time)
(Another 2 seconds for the window to be drawn the first time on mac)
julia> @time plot(1:10,1:10)
0.000763 seconds (2.80 k allocations: 166.023 KiB)Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#178Earlier quoted context omitted.
Regarding 2), code like "for row in pandas.DataFrame(...).iterrows():" where the DataFrame was populated by some array-based reader is creating an indirection. Most good pandas code of course doesn't look like this, but it's regularly unavoidable, hence "There are use cases where either representation is preferred". Without specifying our use case, it is meaningless to talk about "faster" or "slower". For 3) there in…
> there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs" You seem to be under the misapprehension that mainstream CSV libraries parse float values with reduced precision in order to go faster. They do not. A decimal floating-point string represents a precise mathematical value and the only acceptable value for a CSV parser to produce is that value correctly rounded to a Fl…
[deleted]
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#179Julia gets alot of hate nowadays that's for sure https://twitter.com/danluu/status/1315407713430458368?s=19
Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
#180This is great. I'm definitely going to keep Julia in mind the next time my languages self-review comes up, in about 2030. Could even happen as early as 2029.
If only learn Julia in 2029 you will miss out on top notch productivity for another 9 years! Please reconsider for Ur own sake. If not that then for your whisky.