Live data from Hacker News

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

juliacomputing.com

231–236 of 236 posts

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#231
post #97

Earlier quoted context omitted.

>The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue. This is literally the pain point that Julia set out to solve. Nobody uses Python for real programs, they use an ad-hoc kludge of Python and C/C++. This is basically fine if you're doing bread-and-butter things that are well covered by existing libraries, but it's a serious drag if you're trying to do anything really inte…

> Python really isn't keeping up; it was designed by and for hackers What does "hackers" here even mean? Is it just a throw-away disparaging term? Hacker has at least three distinct meanings: deeply knowledgeable software developers, shallowly knowledgeable programmers, and people who break into computer security systems. I assume you are not using the first of these, else you would likely say the Julia developers ar…

>What does "hackers" here even mean? Is it just a throw-away disparaging term?

No, my point is that the Julia team includes a lot of people who aren't primarily software developers. Many of the top committers to Julia are working scientists who have first-hand experience of running HPC jobs with Julia; their input is a crucial factor in the success of Julia as a numerical computing language.

Python is a very good scripting language, but it's not built for numerical computing and it's not built by people who understand the needs of scientific users. It's not a bad tool, it's just the wrong tool for the particular jobs that Julia was built for.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#232
post #97

Earlier quoted context omitted.

> Python really isn't keeping up; it was designed by and for hackers What does "hackers" here even mean? Is it just a throw-away disparaging term? Hacker has at least three distinct meanings: deeply knowledgeable software developers, shallowly knowledgeable programmers, and people who break into computer security systems. I assume you are not using the first of these, else you would likely say the Julia developers ar…

>What does "hackers" here even mean? Is it just a throw-away disparaging term? No, my point is that the Julia team includes a lot of people who aren't primarily software developers. Many of the top committers to Julia are working scientists who have first-hand experience of running HPC jobs with Julia; their input is a crucial factor in the success of Julia as a numerical computing language. Python is a very good scr…

So, "hackers" means people who are "primarily software developers"?

In that case, your statement that Python "was designed by and for hackers" is not correct. You can go back to van Rossum's 1993 paper on Python, at https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.38.... and see:

> Python is used or proposed as an application development language and as an extension language for non-expert programmers by several commercial software vendors. It has also been used successfully for several large non-commercial software projects (one a continent away from the author’s institute). It is also in use to teach programming concepts to computer science students

"non-expert programmers" and "computer science students" are not hackers by your definition, resulting in an internal inconsistency in your claim.

Python indeed wasn't built for doing numerical computing in Python. But it was definitely influenced by the needs of people who 'have first-hand experience of running HPC jobs' - under the "steering" model.

Here's a representative abstract from 1996, quoting https://www.osti.gov/biblio/461415 :

> We have described current approaches and future plans for steering C++ application, running Python on parallel platforms, and combination of Tk interface and Python interpreter in steering computations. In addition, there has been significant enhancement in the Gist module. Tk mega widgets has been implemented for a few physics applications. We have also written Python interface to SIJLO, a data storage package used as an interface to a visualization system named MeshTv. Python is being used to control large-scale simulations (molecular dynamics in particular) running on the CM-5 and T3D at LANL as well. A few other code development projects at LLNL are either using or considering Python as their steering shells. In summary, the merits of Python have been appreciated by more and more people in the scientific computation community.

One of the authors of that presentation, Paul Dubois, was lead developer for Numerical Python, which evolved into NumPy.

The needs of numeric computing, under the steering model, contributed to Python-the-language. Most recently with "@", but also the multiple slice notation a[1:3, 5:11:2] and the change to remove the three-state compare cmp(). Plus under-the-covers support for a wide range of HPC architectures.

In turn, Python's support for steering numeric computing helped it make in-roads in HPC, where it remains to this day.

You may call the steering model 'an ad-hoc kludge of Python and C/C++', but as I've pointed out, you have a tendency to use needless antagonisms, and that's another one of them.

You're the one who claims I don't do "real" programs, only "bread-and-butter things" simply because I don't do bare-metal numeric computing software. Just how much of a multi-disciplinary effort is Julia if bare-metal numeric computing software is the only discipline it focuses on?

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#233

Earlier quoted context omitted.

> Nothing is faster than C that is also high-level. That is myth. C is not necessarily faster on modern hardware, because it does not represent its structure correctly [C Is Not a Low-level Language. Your computer is not a fast PDP-11.]( https://queue.acm.org/detail.cfm?id=3212479 )

Would you sum it up for me and give me examples? Which programming language is generally faster than C that is also high-level? Why do you think people go for C if they want performance? In practice it seems like it is the fastest, popular, stable high-level language out there that has been around for decades. Maybe Forth whose compiler is in assembly, but due to its type system or lack thereof, doubt the same optimi…

> Would you sum it up for me and give me examples?

Sure, subject to pre-payment for my time.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#234

Earlier quoted context omitted.

There is an immense value in most of the ecosystem converging on Python over the last decade and especially with leaving MATLAB behind in the dust. I don't think decade long transitions from old tech to new tech is a bad time scale or even close to that of the churn of web technologies. Julia offers enough of an improvement over Python to warrant a switch over the next 5-10 years and leave Python behind in the same w…

Back 5.5 years ago, I used to complain about the 1 based indexing and the column-major structure of matrices in Julia (both like Fortran), however, those issues have been solved by OffsetArrays and PermutedDimsArrays, giving far more flexibility that is possible in most other languages. It's silly to keep bringing up the issue of one based indexing, when you can use any integer as the base, just like Fortran 90 (so y…

Yes it is possible to make those changes and even if it wasn't it does not detract from the impressive and useful innovations Julia has made. That is why I usually refer to them as cosmetic but that is possibly not the correct word either because it does not convey the implications of having that default beyond simply being unattractive. Cultural may be the correct term.

Julia claims that it wants to solve the two language problem. To do that it must win both the scientific compute, HPC, data science, machine learning, etc. group and the CS, software engineering, dev ops group. I've shown Julia to quite a few members of the latter and they always love the idea and technical capabilities of Julia but visibly cringe when they find out it uses one based indexing. They also almost universally find the syntax ugly.

Unfortunately the available packages to change that also does not help because most people will never change the defaults and IMHO it would be bad to do so. When writing Python you follow the PEP-8 style guide, it is just best practice to follow the languages conventions so everyone is on the same page. This is the same problem with C++ trying to add all these new features and a solid package manager like Rust has, it will never fly, people just won't use it everywhere, it isn't going to be in the standard library, It isn't the default and it isn't part of the culture.

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#235
The problem with CSV files are the endless edge cases.

What I'll typically do for larger ones is that I'll write a program that does the reading and the rudimentary processing in C, and then saves the data as a binary .npy file in /dev/shm (or ImDisk if I'm on windows) and load that up in Python. Massage the data more if necessary and finally do the plotting. I found that this helps, hope it helps you too

Re: CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

#236
post #87

Earlier quoted context omitted.

Right - sounds like you have more of a production support role vs. a data analysis workflow kind of task. Tacking on "exploratory" is helpful but I'm still concerned that you misuse the overall concept of analysis. It's decision-making task, which is practically the opposite of production support.

>Tacking on "exploratory" is helpful but I'm still concerned that you misuse the overall concept of analysis. It's decision-making task, which is practically the opposite of production support. Why should the exploratory and production teams be using completely different tools? That seems like it would cause frictions in productivity and make there be gaps that introduce translation errors. I would venture to say tha…

One is R&D, and one is operations. Obviously there are situations where it makes sense to combine them, but most successful companies with any real technology to sell will not treat R&D this way.
Post reply on HN