I wonder why not much is done to bring high performance scientific computing to common lisp. There are some interesting projects I was able to find like https://github.com/clasp-developers/clasp and https://github.com/marcoheisig/Petalisp and https://github.com/takagi/avm. But I guess it would be good to have a coordinated effort in this area.
Correctness and composability bugs in the Julia ecosystem
281–290 of 419 posts
Re: Correctness and composability bugs in the Julia ecosystem
#282Earlier quoted context omitted.
> Note that Matlab, the workhorse of scientific computing for a few decades now, is even less typed than Julia. You always make this argument when discussing PL features and I find it irksome. People get along fine without this feature, therefore there’s no sense in implementing it. But it cuts the other way, or we’d all still be using assembly. How many Matlab users know things could be better? Was the superiority o…
> People get along fine without this feature, therefore there’s no sense in implementing it. As someone whose job is to add new features to a programming language, that's never been my argument. > But it cuts the other way, or we’d all still be using assembly High-level languages were satisfactorily shown to be more productive than Assembly. I don't claim that no innovation works, just that not all do, and certainly…
> As someone whose job is to add new features to a programming language, that's never been my argument.
I’ve definitely seen you argue along the lines of “it hasn’t been implemented in Java, therefore nobody uses it and we can’t tell if it’s a good idea or not” before. Forgive me for assuming this followed from that.
> You can't come up with a claim, try and fail to support it time and again, and keep asserting it as if it's obviously true, despite the evidence.
But “correctness” of itself is pretty nebulous. If we define it as whether or not the program conforms to one’s intentions with writing it, I would expect static types alone not to show a significant difference in correctness. Probably formal methods do but they have much higher overhead.
However, in terms of eliminating patterns which are literally never correct, like dereferencing null pointers, violating resource lifetimes, or calling methods that don’t exist, static typing can in fact eliminate those patterns.
> the claim that programs in OCaml are more correct than programs in Clojure
My full-time job is Elixir so I know full well the consequences of maintaining large codebases in dynamic languages. I would switch to OCaml in a heartbeat if it ran on the BEAM! I want to know that I am calling functions correctly within a node when the module can be resolved at compile time. This is a really basic thing to want, and not one that dynamic languages can offer. The qualitative difference is similar to that between structured and unstructured programming: I can actually do local reasoning about a function without having to check all the call sites or write a lot of defensive tests.
This is an obvious advantage, and on some level I don’t really care if it contributes to formal correctness or not because it would make my job easier.
Re: Correctness and composability bugs in the Julia ecosystem
#283Wait, are those examples real? I remember complaining about 1-bsaed indexing only to be told "julia is great! we have offsetindex". If it's a source of bugs, that ... greatly reduces my future interest in adopting the language.
I was wondering if the 1-based arrays (and option to change index base) would factor into this. > OffsetArrays in particular proved to be a strong source of correctness bugs. The package provides an array type that leverages Julia’s flexible custom indices feature to create arrays whose indices don’t have to start at zero or one. Array indexing is such a core thing and I don't understand why anything mathematical or…
> Array indexing is such a core thing and I don't understand why anything mathematical or scientific would start with 1.
Counting things is such a core thing to humans that when we have a bunch of N things we think of them as thing #1 to thing #N. We start counting from 1, not 0.
Indexing from 0 in computing is adapting the human mind to the computer, purely for performance reasons that may have been relevant in the 50s or 60s but were beginning to be obsolete by the 70s. It was done so you could access elements of an array by the simplest possible calculation of your offset into heap memory. When your first element is stored at Starting_address, you need i for that first element to be = 0, just so you don't need to have the compiler add another constant term for each element to "Element is at Starting_address + i * sizeof(element)".
Would have been trivial, even then (as Wirth showed) to add that constant term calculation to compilers, but it was done without in C because that eliminated one whole integer operation from each (set of?) array access(es).
In stead, we got the mental gymnastics of
for(i=0, i++, iRe: Correctness and composability bugs in the Julia ecosystem
#284I feel this post is a bit unfair and quite outdated (seems like it's written 9-12 months ago), and I interpret his issue as a prioritization issue, not a language one. If your priorities mandate a more mature ecosystem, you should use one. The Julia ecosystem is much smaller - both in terms of people and development invested, than Python, Java or JavaScript, and still overperforms in many aspects of computing. If those aspects, where Julia is first-of-class, are not your priorities, and your fault tolerance is very low, maybe another tool is better for you.
Also, as every ecosystem, the Julia Ecosystem will naturally see some packages come and go. JSON3 is the third approach to reading JSON (and it's terrific). HTTP.jl is the reference HTTP implementation - Julia hasn't had it's `requests.py` moment. Web frameworks have also been immature, python has had `Django`, `pyramid`, `flask` and so many others before `FastAPI` (along with new language features) came and dominated. Some people need to put effort in attempts that will naturally hit a dead end before we have a super polished and neat FastAPI.jl, and the same goes for everything.
Also, https://github.com/JuliaLang/julia/issues/41096 is referenced with a wrong name that involves the issue's author's misunderstanding, can you update please and, if possible, add a note about the edit?
Re: Correctness and composability bugs in the Julia ecosystem
#285Earlier quoted context omitted.
I think it should be fine for performance AFAIU to use `eachindex` instead; at least I know `eachindex` plays nicely with LoopVectorization.jl with no performance costs there. That said, I think you're exactly right that people may wonder just this and use the seemingly "lower-level" form out of concern with or without testing it.
One of my intentions with the rewrite is to let `@turbo` to change the semantics of "unreachable"s, allowing it to hoist them out of loops. This changes the observed behavior of code like for i = firstindex(x):lastindex(x)+1 x[i] += 2 end where now, all the iterations that actually would have taken place before the error will not have happened. But, hoisting the error check out when valid will encourage people to wri…
Oh, nice!
Re: Correctness and composability bugs in the Julia ecosystem
#286Earlier quoted context omitted.
In particular, not a single issue mentioned in this article would have been prevented by static type checking.
This is not true. For example, the issue regarding custom index ranges causing silent data corruption (6 examples) could be fixed with static types. Look how many of the other bug reports contain the phrase "does not check for" or refer to specific primitive types.
it is only fixed with static typing and no generics (i.e. C/Fortran). If you have a generic array supertype, a statically typed language would let you write exactly the same bug.
Re: Correctness and composability bugs in the Julia ecosystem
#287> Given Julia’s extreme generality it is not obvious to me that the correctness problems can be solved. Julia has no formal notion of interfaces, generic functions tend to leave their semantics unspecified in edge cases, and the nature of many common implicit interfaces has not been made precise (for example, there is no agreement in the Julia community on what a number is). Does all that apply to Python? I think so?…
The main difference between Julia and python is that most of the "core" python ecosystem has had a lot more dev time put into it. Google, Facebook, and Microsoft all have hundreds of full time developers on major python packages.
Re: Correctness and composability bugs in the Julia ecosystem
#288Earlier quoted context omitted.
It just amuses me that one of the big differences between the US and EU is which floor is "first" and which one is "zero" or "minus one".
Yes. A German friend of mine moved into her student dormitory in the US, and when she was told that her room was on the first floor, asked whether there was a lift, because she had a heavy suitcase... Having said that, given that there are basements (in Europe, at least), it makes sense to call the ground floor 0. We are dealing with integers here, not natural numbers.
But floors of multi-storey buildings are a pretty unique exception in the real world in having a characteristic where zero -- the number of stairs you need to climb from the ground floor -- has an actual tangible meaning (on the ground floor).
How many other such examples can you (editorial you; anyone) come up with? Not many, I'd bet.
Re: Correctness and composability bugs in the Julia ecosystem
#289Earlier quoted context omitted.
In particular, not a single issue mentioned in this article would have been prevented by static type checking.
This is not true. For example, the issue regarding custom index ranges causing silent data corruption (6 examples) could be fixed with static types. Look how many of the other bug reports contain the phrase "does not check for" or refer to specific primitive types.
How would static types help with that? Whether your indexing range starts with zero or one or something else isn't necessarily encoded in the type domain. `1:length(A)` is just a range of `Int`s.
Re: Correctness and composability bugs in the Julia ecosystem
#290Earlier quoted context omitted.
But humans don't work 0-based. Try explaining to a bunch of scientists why for rows 2-5 of the DataFrame they have to write df[1:5].
Yeah because humans got it wrong. Really the word for "first" should correspond to the number 0. Try doing a block iteration over an array, or any kind of interval algorithms in 0-based and 1-based. 0-based with right-open intervals just results in way way more elegant, easier to understand and (very) slightly more efficient code.
> ...humans got it wrong. [...] 0-based with right-open intervals just results in way way more elegant, easier to understand...
You must have warped your brain into thinking like a computer (well, a C-family-language compiler) for so long and/or so thoroughly that you no longer think quite like an ordinary human.
Having to sprinkle semi-arbitrary "-1"s all over your code is in no reasonable sense of the words "way way more elegant, easier to understand" than not having to do so.