211–220 of 236 posts
Earlier quoted context omitted.
It's always a transcendental experience to move data from CSV into any more structured/flexible format!
And fast! I got easily 3 orders of magnitude speedup. I used to load CSVs with pandas and do typical filtering operations... oh boy, some of the stuff would take a hole morning, not to mention the memory usage. I do all that in minutes now.
I think there's a strong chance that Swift or Rust take a lot of Python's data science cake. Both of them are extremely fast, concrete, and have lots of investment being poured into numerics and fitting into the Python/ML ecosystem. I don't think Julia or R are going to steal this away. In fact, I think Julia is a major turn off to engineers with some of the bizarre choices they made (eg. 1-based indexing to appease…
Earlier quoted context omitted.
Thanks for the insight. I think still for most engineering development - its mostly the one-off launch time that predominates. This is exactly why I don't use C++ for general purpose use. There is something to be said about how easy Python makes development look - despite of the package management warts. Btw, is there a reason why packages can't be precompiled and even distributed? I am sure you guys have thought abo…
> precompiled and even distributed There is work on this. This can be done with https://julialang.github.io/PackageCompiler.jl/dev/ , but so far only to a rather large binary / “bundle” IIUC, In principle it should be possible to do much better if you knew for sure at compile time which methods you would need to dispatch to for the data you ultimately want to run on.
The reason why we can't distribute precompiled .ji files at a package level is because of the way the package resolver works. It depends on the exact versions of dependencies of every package - and even for the same version of an end-user package, there can be slightly different versions of the dependencies installed depending on constraints imposed by other packages.
One major improvement coming in 1.6 is multi-threaded precompilation, and it will leverage all the cores you have.
> The tools used for benchmarking were BenchmarkTools.jl for Julia, microbenchmark for R, and timeit for Python. Is this a meaningful comparison? These benchmarks repeatedly call the function (with same input file) many times and then take the average of the time spent. However, in real world, we only read a specific file ONCE -- if we call the csv reader several times, it is almost always for different files.
There are two potential issues that this might disregard: 1. cold file cache 2. JIT compile time The cold/hot file cache issue affects all languages equally, so it doesn’t invalidate the comparison. It is possible that CSV file is not in cache and a system's disk I/O is slow enough that it becomes a bottleneck, which would make all parsers equally slow. However, this is not very realistic because CSV files—especially…
Earlier quoted context omitted.
It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7. I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy la…
Right - sounds like you have more of a production support role vs. a data analysis workflow kind of task. Tacking on "exploratory" is helpful but I'm still concerned that you misuse the overall concept of analysis. It's decision-making task, which is practically the opposite of production support.
Why should the exploratory and production teams be using completely different tools? That seems like it would cause frictions in productivity and make there be gaps that introduce translation errors. I would venture to say that just having the exploratory and production teams working using the same code base is a very strong productivity gain, and we've seen this is true in many companies.
Frankly, all these implementations are disappointingly slow. Daniel Lemire and I wrote simdjson to use SIMD to read JSON quickly - CSV is strictly easier than that. I made a start on simdcsv but got bored, but the same principles would apply. IMO this task should be doable at 1-2GB/s on a single core; it's not something that should really need multiple cores.
Your work on simdjson is awesome! Thanks for saving the world so much unnecessary power consumptions :)
> The tools used for benchmarking were BenchmarkTools.jl for Julia, microbenchmark for R, and timeit for Python. Is this a meaningful comparison? These benchmarks repeatedly call the function (with same input file) many times and then take the average of the time spent. However, in real world, we only read a specific file ONCE -- if we call the csv reader several times, it is almost always for different files.
There are two potential issues that this might disregard: 1. cold file cache 2. JIT compile time The cold/hot file cache issue affects all languages equally, so it doesn’t invalidate the comparison. It is possible that CSV file is not in cache and a system's disk I/O is slow enough that it becomes a bottleneck, which would make all parsers equally slow. However, this is not very realistic because CSV files—especially…
Earlier quoted context omitted.
This is also very easy to achieve in Python by using Cython to selectively optimize code. That way for the 99% of code where such optimizations are meaningless and their requirement for static typing is a liability, you can ignore them, and only focus on the 1% of use cases where it matters.
... but Cython is not Python, is it? The argument that "Python can do it, you just have to use something other than Python) is very weird to me. I began looking into Julia because I was forced into Cython during a project and I loathed having to add a separate, statically compiled section of my code that did not integrate well with the rest of Python (e.g. no stack traces through Cython), and kept breaking (because o…
Managing Cython implementations as separate extension modules is a feature, not a bug - it separates concerns and allows only spending any effort on type system constructs in the small amount of cases where those optimizations will help you.
Cython is extremely easy to compile and standard Python packaging tools work easily with it out of the box, conda included.
Earlier quoted context omitted.
> “ Do you really believe that "parse CSV to a data frame" and "parse CSV to a parquet file" and "parse CSV and apply a query to it in-place" are completely separate, unrelated things?“ Yes, absolutely. Very high level design goals are completely different between those use cases and you would be forced to make mutually exclusive trade offs to solve any one of those problems - which will directly lead to concessions…
CSV.jl seems to be proof that you're wrong about that. More concretely, the only difference between parsing to a data frame, parsing to a parquet file, and executing a query over data is a) where you put the data in memory and b) optionally calling some code to operate on the data as you go. It's hard to see how those applications require radically different designs.
Design for general case speed is directly at odds with designing either for executing SQL over top of the file (in which case the trade offs are about indexing the data for record-wise or column-wise operations) or type marshaling for an extremely narrow use case like reading into Spark.
For example, parsing for speed vs parsing for Spark interop are very likely to handle datetime parsing or null parsing significantly differently.
Earlier quoted context omitted.
> “ In Julia, you can write something generic and let the compiler specialize it for you, so you get many different ways of applying the same thing for free.” Python also lets you easily achieve this in many ways - notably fused typing in Cython.
Just adding types doesn't give you that much extra performance in cython (2-3x at most in most cases in my experience) if you want 10x+ speed up you generally have to rewrite your code in a more 'cythonic' way.
With numba it can be even more dramatic. Zero code changes, solely type annotation in the decorator, and you can often get 1000x speedup over pure Python, because numba can optimize away many of the extra CPython data model features that tight loop code may occasionally not need to use.