Live data from Hacker News

Some Insights from a Julia Developer

stochasticlifestyle.com

161–170 of 241 posts

Re: Some Insights from a Julia Developer

#161
post #155

Sometimes, in my darker moments, I have the terrifying thought that one of the reasons that many users like R and made it popular (apart from the historical context of its now many libraries and being the main free version of statistical software), is specifically that it isn't robust and sensibly designed from a programming/analytical perspective. You can download a package, type in a preset command on a preset thin…

I think your fears are misplaced. R users love R because of ease of use. R does sometimes ignores ugly corner cases in favor of that ease of use (though I'm skeptical this damages the validity of the answer in anything like 5% of cases), but that's a side effect. sklearn and pandas are great, but you still simply have to be a programmer to use them, or at least much closer to a programmer than many statisticians want…

In many ways, I think we're actually in agreement :p

Free. Quick. Interactive. 4 lines. Done. What's the value proposition of Julia compared to that? Now you can argue that Julia will allow better type specification and specialisation, etc, etc. But that's not what's valued or used in your example.

Now, my mindset/domain has to engage with such, and i freak out at the implicit-ness/assumptions inherent in such code, especially with some of R's additional design choices.

In my mind, having code that crashes/errors out when something ambiguous/dangerous is encountered in your example a "good thing". But that's not what's valued in the above example.

And while Julia would allow someone to program in such a way as to specialise or raise these errors and concerns, what is valued is the ability to not think about them. And if you don't think about them, or even as a library designer, if you want to program in such a way that your user primarily doesn't want to think about them, then you're effectively very close to reinventing/reimplementing a new "R", with all its costs and benefits.

I'm just making this observation explicit, even though I actually do value the things the Julia crowd seems to value, but I'm not sure I'm in the majority, and I'm not sure you can push that down into the libraries without exposing concerns most users don't want to be exposed to.

Re: Some Insights from a Julia Developer

#162

I really like Julia overall but I'm undecided whether it's good as a general purpose language. Right now I'm working on a few-thousands-lines-of-code project and sometimes wish Julia was more like Swift: - Writing code with Nullables is cumbersome and verbose compared to Swift. - The object.method() notation is sometimes more readable, especially in more complex expression. Plus, in an IDE it works well with completi…

I get what you mean, but I kind of miss the other when working with one. For larger code bases Swift feels "safer", but Julia tends to feel more enjoyable and fast to try out things with.

But yeah I REALLY wish the Julia guys can come up with a nice way of dealing with Nullable.

But I actually don't miss the `object.method()` notation. I find it is so much more straightforward to compose things functionally when everything is a plain function. I feel methods just adds complexity to a language. For readability I tend to use the form `x |> f |> g |> h` if I need to make it easier to read a call like `h(g(f(x)))`.

I agree completion is a bit nicer with `object.method()` but function completion works quite well in the REPL and Juno. I use `methodswith()` to locate relevant functions for a type.

In some ways I think it is actually easier to deal with than in Swift, as Swift base classes have so many methods you can't find the stuff you are interested in quickly.

I have kind of wanted Swift to be the solution for everything, but I see that the API design philosophy for Swift makes it difficult to turn it into a language which is as nice for data science as Julia. There is no focus on making stuff like matrix classes, multidimensional arrays, shell integration etc.

Also I don't quite like that Swift ties me so much to an IDE. Writing Swift code with a plain text editor and a REPL isn't as nice as doing it with Julia.

Re: Some Insights from a Julia Developer

#163

Earlier quoted context omitted.

Not sure how much performance matters here. You're just farming it off to CPLEX or Gurobi anyway. If performance really matters, you'll do this part in C++ anyway.

Unless you're solving toy problems, it matters quite a bit. In mathematical optimization, constructing the problem tends to be just as expensive as solving the problem is – sometimes more so. The existence of expensive commercial systems like AMPL and GAMS that only exist to express optimization problems demonstrates that this is a non-trivial issue that people are willing to pay money for. Using C++ APIs to solvers…

> Unless you're solving toy problems, it matters quite a bit. In mathematical optimization, constructing the problem tends to be just as expensive as solving the problem is – sometimes more so.

With all due respect, this is just false. I can construct a million dimensional linear program for a large network problem in a few seconds in python. Solving it with cplex can take minutes.

I have a Ph.D. in computational mathematics and do this for a living. I write lots of C, and still almost exclusively use Python or Lua for problem formulation because the performance just doesn't matter.

I'm certain there are some cases where it's true. One I've encountered is doing real-time solution of MILP for adaptive path planning. Here the problems are of moderate size and you want to reconstruct them from geometry data every 10ms or so. Here Python will indeed bite you in the problem formulation stage.

Re: Some Insights from a Julia Developer

#164

It's great and all, but I can't justify switching languages for minor improvements over Python + numpy/scipy. I'd be abandoning: * My deep knowledge and experience with Python * My entire codebase * The ability to work on projects with colleagues who don't also switch * The certainty that when I leave my current job, someone will be able to pick up after me * Zero-based indexing I've started to do some work in Rust w…

You will want to switch as soon as you have to confront the Global Interpreter Lock (GIL)

I've confronted it plenty of times, and for my use cases it's not really an issue. Many numpy functions release the GIL, so by keeping my performance-critical stuff in numpy it's just not an issue for me. I sometimes split tasks up with multiprocessing, which is a GIL workaround to be sure, but the memory footprint of extra interpreters is negligible compared to the data I work with.

Re: Some Insights from a Julia Developer

#165
post #155

Earlier quoted context omitted.

I think your fears are misplaced. R users love R because of ease of use. R does sometimes ignores ugly corner cases in favor of that ease of use (though I'm skeptical this damages the validity of the answer in anything like 5% of cases), but that's a side effect. sklearn and pandas are great, but you still simply have to be a programmer to use them, or at least much closer to a programmer than many statisticians want…

In many ways, I think we're actually in agreement :p Free. Quick. Interactive. 4 lines. Done. What's the value proposition of Julia compared to that? Now you can argue that Julia will allow better type specification and specialisation, etc, etc. But that's not what's valued or used in your example. Now, my mindset/domain has to engage with such, and i freak out at the implicit-ness/assumptions inherent in such code,…

DWIM (Do What I Mean) at the UI level seems a good feature to aspire to. The sanity check could (and should) be done using an independent method (say manually check a couple of things, or is that too out-of-dated?).

Re: Some Insights from a Julia Developer

#166

It's great and all, but I can't justify switching languages for minor improvements over Python + numpy/scipy. I'd be abandoning: * My deep knowledge and experience with Python * My entire codebase * The ability to work on projects with colleagues who don't also switch * The certainty that when I leave my current job, someone will be able to pick up after me * Zero-based indexing I've started to do some work in Rust w…

The improvements are not minor they are massive. Citing rust shows that the advantage of Julia has not been explained well enough. Julia allows you to write as performant code as Rust with a much smaller investment in learning. You cite your concern for spending time learning something new. That makes no sense considering the high learning curve and complexity of Rust compared to Julia. Julia is quite fast to learn a…

I've been using Julia for a year or two now, and over that time my enthusiasm for the language has waned, not grown.

Here's the standard pitch for Julia: C-like performance with R/Python/MATLAB-like expressivity.

This is undeniably true. For someone coming from R/Python/MATLAB, you will be able to program as easily in Julia, if not more easily and more cleanly, and get much better speed.

However, what I've come to realize now is that you can get something similar from Kotlin, Nim, and Crystal, but those are much more general-purpose, and often are more consistent in their performance. They don't have the same numerical libraries at the moment, but they have a much bigger use-case target, and for something like Kotlin, a lot more leverage from the community.

Rust is a lot lower-level, but as the numerical resources grow (and they inevitably will) and layers of abstraction are added, what you will probably get is something very similar to Julia in expressivity but more extensible downward, and probably more performant. Rust has a ton of room to grow, and has many good role models from C (e.g., Eigen).

This also isn't even touching on PyPy and things of that sort.

I like Julia and still reach for it when I'm starting a project. Often it's worked great, but other times I have to go back to R or Python for some library that calls C, because the Julia library, while usable in a way that pure R or Python isn't, is still not actually as fast as C. Sometimes "on the order of magnitude of C" is not actually good enough, and you need C. Also, if you poke around for benchmarks that aren't posted by the Julia developers themselves, you'll see different conclusions about the real-world performance of Julia, so there's that.

I guess I read articles like the linked post, and I end up feeling like they're misleading. Is Julia better for data analysis than R or Python? Yes, almost certainly if you were starting from scratch and not calling any products of outside languages. Is Julia better than R or Python + C as it is in the real world? Not sure, because you still have to call C anyway at some point if you're being honest. Is it better than Kotlin, Nim, or Rust? I'm even less sure. If Julia wants to maintain long-term competitiveness, it has to reach out in the opposite direction of those languages, from numerics -> general purpose computing, because they are inevitably moving from general purpose computing -> numerics, and they are bringing a lot with them.

Re: Some Insights from a Julia Developer

#167

It's great and all, but I can't justify switching languages for minor improvements over Python + numpy/scipy. I'd be abandoning: * My deep knowledge and experience with Python * My entire codebase * The ability to work on projects with colleagues who don't also switch * The certainty that when I leave my current job, someone will be able to pick up after me * Zero-based indexing I've started to do some work in Rust w…

Zero based indexing? I find it not too bad to switch between languages that use zero & one based indexing.

It's not that one is incapable of doing it. It's that one way is wrong and the other is right!

Re: Some Insights from a Julia Developer

#168

It's great and all, but I can't justify switching languages for minor improvements over Python + numpy/scipy. I'd be abandoning: * My deep knowledge and experience with Python * My entire codebase * The ability to work on projects with colleagues who don't also switch * The certainty that when I leave my current job, someone will be able to pick up after me * Zero-based indexing I've started to do some work in Rust w…

Zero based indexing? I find it not too bad to switch between languages that use zero & one based indexing.

I don't have problems with it, but I also never had an issue with Python's whitespace, or languages without semicolons, or languages which prefer to use begin/end blocks. I don't get why it's a big deal.

The one exception to that is when switching between languages. Sometimes one forgets which language one is in and a syntax error will result. Like using the dot operator to concatenate strings in Python or leaving semicolons off the end of PHP statements.

Re: Some Insights from a Julia Developer

#169

Earlier quoted context omitted.

Zero based indexing? I find it not too bad to switch between languages that use zero & one based indexing.

It's not that one is incapable of doing it. It's that one way is wrong and the other is right!

Right, so zero-based is wrong unless one is using a language close to assembly.

Re: Some Insights from a Julia Developer

#170

Earlier quoted context omitted.

For values with compiler-visible scoped lifetime, the compiler will automatically promote them to stack variables. There is currently no way to enforce this happening, but it would be perfectly possible to add such an annotation. Regarding preallocation, you tend to want to avoid dynamic memory allocation in high performance applications anyway, so whether you do that in C++ or in Julia, doesn't really make too much…

> Regarding preallocation, you tend to want to avoid dynamic memory allocation in high performance applications anyway, so whether you do that in C++ or in Julia, doesn't really make too much of a difference. Yes! Which is why it shouldn't take me digging through 3 different documents and still getting it partly wrong! C/C++/Rust make this easy by annotating the type modifiers(&/*/Box/etc) with how the object lives i…

You just do @. and you're done?
Post reply on HN