In terms of saving human time I have found R to be fastest (in human time) for iterative prototyping, exploring, and visualising data
R still has the best statistical package ecosystem, although python is catching up.
Correctness and composability bugs in the Julia ecosystem
51–60 of 419 posts
Re: Correctness and composability bugs in the Julia ecosystem
#52I've spent a lot of time developing large computational codebases in Julia, and I think the most insidious of these issues is a product of no formal way of enforcing interfaces. Using one of the common packages to build a trait system and add some sort of guarantee that all the right methods are implemented for a given trait simplifies maintenance dramatically.
This doesn't catch mathematical bugs, but those crop up everywhere. Instead, knowing what the interfaces must be specified so you can trust your implementation is crucial, and being able to know when it is invalidated is invaluable.
I've had a few awful bugs involving some of the larger projects in this language, but a proper interface/trait system would simplify things exponentially. There are some coding style things that need to be changed to address this, like using `eachindex` instead of `1:length(A)` for array iteration as the example in the article points out. However, these should be one-off lessons to learn, and a good code linter should be able to catch potential errors like this.
Between a good code linter (or some static analysis, I'm pulling for JET.jl) and a formal interface spec, I really think most of Julia's development-side issues could be quelled.
Re: Correctness and composability bugs in the Julia ecosystem
#53Quoted post unavailable.
Genuinely curious (I don’t work with low level systems day-to-day), but why would C/C++ be better for safety? I would imagine that the risk of bugs related to memory access and data racing would be pretty high, compared to a higher-level language or something like Rust that focuses on safety.
AFAIK (and I am not an expert) when they use C/C++ in automotive systems (in the automotive part, not elsewhere) they might tend to use deterministic programming (f.e. fixed-length arrays, no allocations, etc.) which eliminate most of the bugs.
But, again, I am not an expert and Toyota's sudden acceleration problem was said to be caused by bad C programming (said by some, not by others).
Re: Correctness and composability bugs in the Julia ecosystem
#54If you look at the history of lots of packages in matlab they fixed tons of bugs that sound similar to this stuff over the years. It requires consistent hard work by a core group of people who understand the issues to get everything right. I have no idea who maintains Julia and these packages but the author of the article mentions this as language problems — aren’t these just bugs? Like if gcc was incorrectly multipl…
Julia has more than 18k closed issues on its github. No wonder such an active user encountered a lot of it. It's not a problem with the language, though. Yes, it allows to use offsetarrays and @inbounds together, but C can read out-of-memory locations too, so what?
Edit: Julia is better than C in this regard, since the usage of @inbounds is explicit, i.e. everyone can see that the code is potentially unsafe.
Re: Correctness and composability bugs in the Julia ecosystem
#55The power of allowing everyone to make foundational types and functions that work together is indeed dangerous. I'm not sure you are better off in the even more dangerous waters of c/c++/fortran, except that they are older and more established with many times the man-hours sunk into them. Is there a good way to control the interaction of these many different libraries with losing the generality and composability of Julia?
I will say that as a matter of language design 1 based indexing is perfectly fine, 0 based indexing is perfectly fine. Choose your own indexing is a hilarious foot gun, so no surprise it went off sometimes. Fortunately using it seems to be quite rare.
Re: Correctness and composability bugs in the Julia ecosystem
#56Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…
Good comments, Chris. I think the author has a little bit of nuance in that Julia isn't correct in the specific use cases he needs them to be. While your point is also well taken that Julia is correct in cases where other languages aren't as well.
I'm a little unfamiliar with the versioning in the package ecosystem, but would you say most packages follow or enforce SemVer? Would enforcing a stricter dependency graph fix some of the foot guns of using packages or would that limit composability of packages too much?
Re: Correctness and composability bugs in the Julia ecosystem
#57Earlier quoted context omitted.
I was wondering if the 1-based arrays (and option to change index base) would factor into this. > OffsetArrays in particular proved to be a strong source of correctness bugs. The package provides an array type that leverages Julia’s flexible custom indices feature to create arrays whose indices don’t have to start at zero or one. Array indexing is such a core thing and I don't understand why anything mathematical or…
> Array indexing is such a core thing and I don't understand why anything mathematical or scientific would start with 1. Because that's how maths work? Literally everywhere in maths you count from 1, except in software engineering. That's why. I hope that clarified your confusion.
My hot take is that 1-based indexing is often a mistake in math too. It's also not universal, even within math. And linear algebra doesn't need 1-based indexing either, and some operations are even more easily expressed with 0-based indexing.
Re: Correctness and composability bugs in the Julia ecosystem
#58I mean this looks like good potential targets to improve the language moving forward, it's healthy to not be in awe of your tools and push to make them better. I don't see this as "bad" honestly.
It seems like the point of the article is that that push is insubstantial, if it even exists. Given the language has been around this long it's a bit worrying that stuff like that is the potential target for moving a language forward. Julia has always had a reputation in my mind at least of being "by academics, for academics" and there's unfortunately a dark side to that in terms of reliability and maintainability. T…
For what it's worth many people feel similarly about R. R is great for people actively working in statistics research (I assume because that's what I'm always told). But for a lot of us who just want to do some analysis, it's constantly breaking and we've learned to default to just starting from scratch when we need to revisit something we did a few years ago. Or we figure out how to buy a commercial system.
Re: Correctness and composability bugs in the Julia ecosystem
#59Earlier quoted context omitted.
I was wondering if the 1-based arrays (and option to change index base) would factor into this. > OffsetArrays in particular proved to be a strong source of correctness bugs. The package provides an array type that leverages Julia’s flexible custom indices feature to create arrays whose indices don’t have to start at zero or one. Array indexing is such a core thing and I don't understand why anything mathematical or…
> Array indexing is such a core thing and I don't understand why anything mathematical or scientific would start with 1. Because starting with 0 is neither math nor array indexing in general. It's just how the base addresss of an array pointer memory block was referenced in C (and it spread from there). Which is why all math focused languages use 1-based (fortran, apl, matlab, r, mathematica, etc.)
> Because starting with 0 is neither math nor array indexing in general.
It very, very much is. Polynomials all start at a zero "index", as does just about every expansion I can think of (Fourier, Bessel, Legendre, Chebyshev, Spherical Harmonic, etc.) Combinatorics, too, make lots of use of zero indices and zero-sized sets. As for arrays, I'll leave it to Dijkstra[1] to explain why zero indexing is most natural. Zero indexing overwhelmingly makes the most sense in both math and computers because indexing is a different operation than counting.
[1]: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/E...
Re: Correctness and composability bugs in the Julia ecosystem
#60> If you pass it an array with an unusual index range, it will access out-of-bounds memory: the array access was annotated with @inbounds, which removed the bounds check. It think making indexes configurable is a huge mistake. Even if they are not ideal for the situation, having a single way to do indexes makes a huge source of confusion and potential bugs just go away. And this is orthogonal to whether you pick 0 or…
Indexes being configurable makes a ton of sense. It's why so many languages end up with a slice type (or array_view or span or whatever you want to call it). Why shouldn't the base array type just itself be the slice type?