Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Correctness and composability bugs in the Julia ecosystem

yuri.is

Post reply on HN
Print view View original

371–380 of 419 posts

Re: Correctness and composability bugs in the Julia ecosystem

#371

Post by jrochkind1 »

cbkeller wrote: post #333
josephg wrote: post #332

Earlier quoted context omitted.

As a user, I’d prefer “correct but lacking composability” over “composable but sometimes my results will be silently wrong”. What is Julia’s composability useful for if it leaves me unable to trust my results?

The answer as far as I can see is to write integration tests when you want to use composability (and/or just check that the packages in question already have integration tests against each other -- increasingly many do). It's not especially hard or anything, but you need to know to do it.

Hm. Some cases are where non-composable things might just fail; that is indeed "not especially hard or anything" to detect with simple integration tests, sure. But those you'll also probably notice without integration tests too, when your code just obviously doesn't work. Either way, sure, you'll find out about this quick, and know you have to fix it if you want to use that code to get the results you wanted to get.

But some of the examples in OP of non-composability are where edge cases (not all cases) will give you the wrong answers. In the worst case, in ways that aren't immediately obvious.

I think it's too optimistic to say that you just need to know that you should write some integration tests and then write some easy and obvious ("not especially hard") ones, and then you'll always catch these easily and early.

Re: Correctness and composability bugs in the Julia ecosystem

#372

Post by cestith »

cb321 wrote: post #214
cestith wrote: post #47

Earlier quoted context omitted.

If you really want safety, use Rust or Ada.

Or Nim [1]..kind of an Ada with Lisp macros and more Pythonesque surface syntax. All three are ahead-of-time compiled/less REPL friendly than Julia, though. Taking more than 100 milliseconds to compile can be a deal breaker for some, especially in exploratory data analysis/science settings where the mindset is "try out this idea..no wait, this one..oops, I forgot a -1" and so on. In my experience, it's unfortunately…

The Python-like syntax is a minus for me. I didn't care for semantic leading whitespace in ABC. I don't care for it in Python. I therefore doubt I'd care for it in Nim. If that's your thing, an Ada with Lisp-like macros sounds delightful.

Re: Correctness and composability bugs in the Julia ecosystem

#373

Post by pron »

jolux wrote: post #369
pron wrote: post #357

Earlier quoted context omitted.

> to know whether it’s the right tool for anything and whether strong types are actually a good idea. Nope. I don't know what you mean by "a good idea", and I prefer typed languages myself (mostly for tooling support), but I do often point out that the claim that types improve correctness — let alone the claim that they do that better than other approaches — is an empirical claim that is not supported by empirical ev…

> Types certainly eliminate various kinds of errors, yet studies did not find that they reduce bugs Right, but this is why I asked about structured programming. These things are hard to study by their nature. I understand there’s not a consensus on this, but there are a lot of programmers who feel quite strongly that static types reduce bugs. Maybe that’s not good enough for you! But it’s clear that you accept other…

> These things are hard to study by their nature.

Small effects are hard to study by their nature. Big effects are usually easy to spot.

> there are a lot of programmers who feel quite strongly that static types reduce bugs

There are a lot of people who feel quite strongly that homeopathy cures all kinds of diseases, but they've failed to demonstrate that.

> But it’s clear that you accept other practices as beneficial on insufficient evidence

It's not about "accept." I myself practice typed programming without asserting the empirical claim that it reduces bugs, that seems not to be true (or at least, the effect does not seem to be big).

> The question is not whether static types prevent the most common types of bugs, or the most dangerous

So far, we've failed to show that static types reduce bugs, period. You're allowed to like them and promote them, but your feeling towards them does not make a specific empirical claim more or less true.

Re: Correctness and composability bugs in the Julia ecosystem

#374

Post by cbkeller »

jrochkind1 wrote: post #371
cbkeller wrote: post #333

Earlier quoted context omitted.

The answer as far as I can see is to write integration tests when you want to use composability (and/or just check that the packages in question already have integration tests against each other -- increasingly many do). It's not especially hard or anything, but you need to know to do it.

Hm. Some cases are where non-composable things might just fail; that is indeed "not especially hard or anything" to detect with simple integration tests, sure. But those you'll also probably notice without integration tests too, when your code just obviously doesn't work. Either way, sure, you'll find out about this quick, and know you have to fix it if you want to use that code to get the results you wanted to get.…

Sure, but that's a caveat that applies to testing in general, not something somehow special to composability. Bugs that only appear in edge cases or give wrong answers are always a concern when writing tests, that's nothing new.

I don't actually even see any evidence here that these sorts of bugs are more common or more insidious than any other sort of bug, rather it looks to me that since dispatch-based composability is a bit of a new phenomenon, people haven't always been on the lookout for these bugs yet, or learned the patterns of where and what to test for to find them -- but once you've seen a few cases, patterns start appearing and it becomes more clear what you need to test, just like anything else.

The broader issue to me is that I think people often underestimate the degree to which Julia is a fundamentally different programming paradigm than any other language most people have used before (unless you've spent a lot of time in maybe CLOS or Dylan) -- there's not even an agreed-upon name for it, but I'd call it "dispatch-oriented programming". Often times people will come in to Julia expecting all their expertise and intuition from class-based OO or whatever other paradigm to just effortlessly transfer to Julia, and that's really not the case, which tends to lead to frustration.

Re: Correctness and composability bugs in the Julia ecosystem

#375

Post by pron »

ThenAsNow wrote: post #366
pron wrote: post #358

Earlier quoted context omitted.

"Awesome", i.e. more enjoyable for you, and "more correct", i.e. fewer bugs in production, are two very different things. I also prefer typed languages for the software I tend to write and find them more enjoyable, but that still doesn't make me claim that types lead to more correct software.

I am not familiar with the studies you are relying on to make the point that statically-typed languages have no significant difference in terms of number of bugs in production compared to dynamically-typed. Measuring such things is challenging, and the most useful measure may not be in terms of "bugs in production" but by a number of other measures, such as how long it takes to surface bugs after the code is accepted…

Anyone is allowed to prefer a programming style that suits their aesthetics and habits, and like that one over all others. Aesthetic preferences are a very valid way to choose your programming language — ultimately that's how we all pick our favourite languages — and there's no need to make up universal empirical claims to support our preferences.

Here's a good talk to watch on the subject: https://youtu.be/ePCpq0AMyVk

And here's a summary of various studies done: https://danluu.com/empirical-pl/

As of today, what we know is that if there's a positive effect of types on correctness, then it is probably a small one.

There's really no need to assert what is really a conjecture, let alone one that's been examined and has not been verified. If you believe the conjecture is intrinsically hard to verify, you're conceding that you're only claiming a small effect at best (big effects are typically not hard to verify), and so there's even less justification for continuing to assert it. It's okay to prefer typed languages even though they do not, as far as we know, have a big impact on correctness.

Re: Correctness and composability bugs in the Julia ecosystem

#376

Post by jrochkind1 »

cbkeller wrote: post #374
jrochkind1 wrote: post #371

Earlier quoted context omitted.

Hm. Some cases are where non-composable things might just fail; that is indeed "not especially hard or anything" to detect with simple integration tests, sure. But those you'll also probably notice without integration tests too, when your code just obviously doesn't work. Either way, sure, you'll find out about this quick, and know you have to fix it if you want to use that code to get the results you wanted to get.…

Sure, but that's a caveat that applies to testing in general, not something somehow special to composability. Bugs that only appear in edge cases or give wrong answers are always a concern when writing tests, that's nothing new. I don't actually even see any evidence here that these sorts of bugs are more common or more insidious than any other sort of bug, rather it looks to me that since dispatch-based composabilit…

Right, it's a caveat to testing in general that shows "well, just write integration tests, it's not especially hard to do" is not a satisfactory solution to the problems discussed in OP with specific actually occuring examples. You were the one suggesting it was, not me!

Re: Correctness and composability bugs in the Julia ecosystem

#377

Post by cbkeller »

jrochkind1 wrote: post #376
cbkeller wrote: post #374

Earlier quoted context omitted.

Sure, but that's a caveat that applies to testing in general, not something somehow special to composability. Bugs that only appear in edge cases or give wrong answers are always a concern when writing tests, that's nothing new. I don't actually even see any evidence here that these sorts of bugs are more common or more insidious than any other sort of bug, rather it looks to me that since dispatch-based composabilit…

Right, it's a caveat to testing in general that shows "well, just write integration tests, it's not especially hard to do" is not a satisfactory solution to the problems discussed in OP with specific actually occuring examples. You were the one suggesting it was, not me!

Well I don't think it's hard any more than writing any other sort of test in a given language. That doesn't mean it doesn't require expertise in that language.

Re: Correctness and composability bugs in the Julia ecosystem

#378

Post by btilly »

ChrisRackauckas wrote: post #354
patrickkidger wrote: post #351

Earlier quoted context omitted.

100% this. In a discussion on "cultural correctness issues prevents me from using Julia", it's very telling that the response is "more speed!" There's been a decent number of posts based around "Julia has these problems". And I don't think that's because the world at large has a vendetta; I think it's because the world at large desperately wants to use Julia, but struggle with hard blocks that are currently preventin…

Two of the mentioned packages, Lux and Enzyme, have increased correctness and decreased API surface... and were not mentioned for speed (though a lot of things end up faster when it's easier to prove correctness in the compiler)... so the response wasn't "more speed" but "here's correctness with resulting speed"...

Actually Enzyme was mentioned for speed, not correctness. To verify, go back and see that you wrote, Pretty much everything is switching to Enzyme which improves performance quite a bit over Zygote...

You didn't mention speed on Lux, but it is a rewrite. The rule is that a rewrite should be assumed to be buggy until proven otherwise. A culture of having everything permanently under construction comes with upsides and downsides. And unless you have good testing, correctness problems is one of the downsides.

Re: Correctness and composability bugs in the Julia ecosystem

#380

Post by jolux »

pron wrote: post #373
jolux wrote: post #369

Earlier quoted context omitted.

> Types certainly eliminate various kinds of errors, yet studies did not find that they reduce bugs Right, but this is why I asked about structured programming. These things are hard to study by their nature. I understand there’s not a consensus on this, but there are a lot of programmers who feel quite strongly that static types reduce bugs. Maybe that’s not good enough for you! But it’s clear that you accept other…

> These things are hard to study by their nature. Small effects are hard to study by their nature. Big effects are usually easy to spot. > there are a lot of programmers who feel quite strongly that static types reduce bugs There are a lot of people who feel quite strongly that homeopathy cures all kinds of diseases, but they've failed to demonstrate that. > But it’s clear that you accept other practices as beneficia…

> There are a lot of people who feel quite strongly that homeopathy cures all kinds of diseases, but they've failed to demonstrate that.

This is a really poor comparison. How many people who know enough to judge these things believe that? I can understand that you’re rigorous in the empirical claims you accept here, but surely you can see that experienced software developers have somewhat more basis to make claims about type systems than random people do about homeopathy.

> or at least, the effect does not seem to be big

Fair enough! It’s probably not as big as testing or code review but I do think it exists. You mentioned the TypeScript study, it’s not like there’s no evidence for believing this like there is with homeopathy.

> So far, we've failed to show that static types reduce bugs, period.

I mean, how could they not? You still haven’t explained this part, except with vague allusions that the problems they catch are not the most common problems. I’ve explained how static types reduce bugs: by eliminating cases where you intend to handle a case and forget to. I’ll specify that I mean a type system with exhaustivity checking or narrowing with control flow (like TS or Kotlin). This is not an empirical argument, it’s a rational argument, and I still legitimately don’t understand what the flaw is. The way I see it, static type systems restrict operations on types to those known to be valid at compile time. In a sound type system and compiler, the resulting program is guaranteed not to make invalid operations on types at runtime. Dynamic programs are fully free to make invalid operations on types at runtime. Some number of bugs are caused by making invalid operations on types at runtime. Where are those bugs in a statically typed program, if they haven’t been eliminated?

Post reply on HN