Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Correctness and composability bugs in the Julia ecosystem

yuri.is

Post reply on HN
Print view View original

171–180 of 419 posts

Re: Correctness and composability bugs in the Julia ecosystem

#171

Post by FabHK »

dekhn wrote: post #137
throwawaymaths wrote: post #98

Earlier quoted context omitted.

It causes problems for people with a CS background too. I once numbered machines in racks with zero-indexing (so that they could match up with zero-indexed ip addresses). Even though literally everyone who touched those machines had CS background: DO NOT DO THIS.

It just amuses me that one of the big differences between the US and EU is which floor is "first" and which one is "zero" or "minus one".

Yes. A German friend of mine moved into her student dormitory in the US, and when she was told that her room was on the first floor, asked whether there was a lift, because she had a heavy suitcase...

Having said that, given that there are basements (in Europe, at least), it makes sense to call the ground floor 0. We are dealing with integers here, not natural numbers.

Re: Correctness and composability bugs in the Julia ecosystem

#172

Post by mbauman »

kazinator wrote: post #166
ChrisRackauckas wrote: post #44

Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…

> I agree, code should never do that. It should be `eachindex(A)` Will that generate the same code as "i in 1:length(A)"? Maybe whoever wrote that didn't believe so at least, or perhaps didn't find it so at the time. The reason @inbounds would have been used is performance, so that's likely why the for loop header was written that way?

`eachindex` is — in quite a few situations — faster than `1:n`.

We've also been trying to promote a culture of not blindly putting `@inbounds` notations on things as the compiler gets smarter. `@inbounds` is a hack around a dumb compiler, especially when the loop is as simple as many of these examples. It's not needed there anymore (but was 5 years ago).

Re: Correctness and composability bugs in the Julia ecosystem

#173

Post by cbkeller »

kazinator wrote: post #166
ChrisRackauckas wrote: post #44

Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…

> I agree, code should never do that. It should be `eachindex(A)` Will that generate the same code as "i in 1:length(A)"? Maybe whoever wrote that didn't believe so at least, or perhaps didn't find it so at the time. The reason @inbounds would have been used is performance, so that's likely why the for loop header was written that way?

I think it should be fine for performance AFAIU to use `eachindex` instead; at least I know `eachindex` plays nicely with LoopVectorization.jl with no performance costs there.

That said, I think you're exactly right that people may wonder just this and use the seemingly "lower-level" form out of concern with or without testing it.

Re: Correctness and composability bugs in the Julia ecosystem

#174

Post by grumpyprole »

ChrisRackauckas wrote: post #44

Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…

> Everything has correctness issues somewhere.

Yes but Julia is (yet another) dynamic language, presumably for "ease of use". A language with static types would have made it easier to build correct software (scientific code in e.g. OCaml and F# can look pretty good). Julia chose a path to maximize adoption at the expense of building a reliable ecosystem. Not all languages choose to make this trade-off.

Re: Correctness and composability bugs in the Julia ecosystem

#175

Post by snicker7 »

cozzyd wrote: post #163
harpiaharpyja wrote: post #63

Earlier quoted context omitted.

I'm guessing you've never used Python for a serious project before, because your statement is incorrect. Aside from the fact that Python is memory safe, my experience has been that it is far easier to avoid logic errors in Python than in C/C++. Equivalent Python code is shorter and less noisy than C/C++, and Python makes it easier to create succinct and simple to use abstractions. The result is that code is far easie…

In my experience it's much harder to write Python that won't crash than C/C++ that won't crash. With Python, rarely taken code paths can have very dumb errors in them (e.g. accidentally introducing a new variable with a different name than the variable you wanted or typoing a method name, or passing arguments with the wrong type) that would get caught by a C/C++ compiler but won't be caught in Python until the crash…

C/C++ memory errors are literally responsible for 70% of ALL critical security bugs. Most python "dumb errors" can be caught by linters. Memory is infinitely harder.

Source -- https://msrc-blog.microsoft.com/2019/07/16/a-proactive-appro...

Re: Correctness and composability bugs in the Julia ecosystem

#176

Post by kazinator »

mbauman wrote: post #172
kazinator wrote: post #166

Earlier quoted context omitted.

> I agree, code should never do that. It should be `eachindex(A)` Will that generate the same code as "i in 1:length(A)"? Maybe whoever wrote that didn't believe so at least, or perhaps didn't find it so at the time. The reason @inbounds would have been used is performance, so that's likely why the for loop header was written that way?

`eachindex` is — in quite a few situations — faster than `1:n`. We've also been trying to promote a culture of not blindly putting `@inbounds` notations on things as the compiler gets smarter. `@inbounds` is a hack around a dumb compiler, especially when the loop is as simple as many of these examples. It's not needed there anymore (but was 5 years ago).

The question is: is it at least as fast in all situations? Was it always that way?

The 1 to length loop just has to initialize a local variable and step it; it cannot do anything else. It doesn't worry about the kinds of array that A may be, with its particular configuration of indexing, right?

You may promote a culture of not doing certain things, but that by itself won't make those things disappear from existing code.

Say you're trying to ship some product and you receive a bulletin from the language mailing list encouraging you, "try not to use @inbounds, it's a hack around a dumb compiler". You know you have that in numerous places; but you're not going to stop what you're doing and start removing @inbounds from the code base. If you're remarkably conscientious, you might open a ticket for that, which someone will look into in another season.

Re: Correctness and composability bugs in the Julia ecosystem

#177

Post by lostmsu »

cs702 wrote: post #115

A more appropriate title for the OP would have been: "A new language that makes it easy to write and use generic algorithms on a growing number of custom types developed by others is bound to experience growing pains as difficult-to-foresee correctness bugs have to be discovered and fixed over time." In my humble opinion, this kind of universal composability , which Julia makes easy via multiple dispatch and naming c…

I am not sure I know of any statically typed languages with generics, that experienced the same kind of problems on multiple occasions. The only one I am aware of is C# and array variance, which is kept for compatibility purposes.

Re: Correctness and composability bugs in the Julia ecosystem

#178

Post by rcthompson »

ChrisRackauckas wrote: post #44

Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…

Is "for i in 1:length(A)" ever correct? Should Julia just emit a warning any time it encounters that pattern? Or maybe something slightly more complicated, such as that pattern followed by usage of i to index into A inside the loop?

Re: Correctness and composability bugs in the Julia ecosystem

#179

Post by mbauman »

ChrisRackauckas wrote: post #44

Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…

I do think there's a particularly unique challenge to Julia in that so many packages can theoretically coexist and interoperate. While it quadratically increases the power of Julia, it also quadratically increases the surface area for potential issues. That — to me — is the most interesting part of the blog post. How can we help folks find the "happy" paths so they don't get lost in the weeds by trying to differentiate a distributed SVD routine of an Offset BlockArray filled with Unitful Quaternions? And — as someone who worked with and valued Yuri's reported issues and fixes — how can I more quickly identify that they're not someone who gets joy out of making such a thing work?

Re: Correctness and composability bugs in the Julia ecosystem

#180

Post by pron »

grumpyprole wrote: post #174
ChrisRackauckas wrote: post #44

Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…

> Everything has correctness issues somewhere. Yes but Julia is (yet another) dynamic language, presumably for "ease of use". A language with static types would have made it easier to build correct software (scientific code in e.g. OCaml and F# can look pretty good). Julia chose a path to maximize adoption at the expense of building a reliable ecosystem. Not all languages choose to make this trade-off.

> A language with static types would have made it easier to build correct software

This claim is repeated often, but numerous attempts have failed to demonstrate that this is generally the case in practice (there have been a couple of studies showing an effect in very specific circumstances). Static types might indeed assist with correctness, but they are not the only thing that does, and in some situations they could come at the expense of others. I.e., even if types were shown to significantly help with correctness, it does not follow that if you want correctness your best course would be to add types.

Given empirical studies, the current working hypothesis should be that if static types do have a positive effect on correctness, it is a small one (if it were big, detecting it would have been easy).

Note that Matlab, the workhorse of scientific computing for a few decades now, is even less typed than Julia. That's not to say that Julia doesn't suffer from too many correctness issues (I have no knowledge on the matter), but even if it does, there is little support for the claim that typing is the most effective solution.

Post reply on HN