Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

dev-discuss.pytorch.org

Post reply on HN
Print view View original

211–220 of 291 posts

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#211

Post by jhgb »

freemint wrote: post #198
jhgb wrote: post #153

Earlier quoted context omitted.

> I've never seen recursive looping over a thing get turned into efficient SIMD code in any language. Starting with loops has no good reason to be better able to achieve that but for practical compilers it makes a huge difference. Well, for example at the very least in Common Lisp you'll have much more joy with higher-order functions than with loops. The simple reason for that is the existence of compiler macros ( ht…

Maybe loops are not fun in common lisp the? I had much fun with loops in PicoLisp. Julia also has similar macro capabilities to common lisp. > And it's much easier to figure out what the function composition does than to write a loop vectorizer I probably agree but complicated loop vectorisation would probably expressed in Einstein tensor contraction notation such as Tullio.jl provides. I wonder a Fourier transform w…

Loops are a lot of fun in Common Lisp, what with LOOP (https://gigamonkeys.com/book/loop-for-black-belts.html), ITERATE (https://common-lisp.net/project/iterate/doc/Don_0027t-Loop-I...), and FOR (https://shinmera.github.io/for/). But their fundamental problem of trying to extract a "bird's eye view" of what you're actually trying to do remains here.

Maybe a new iteration construct specifically supporting parallelization would help here. Some of these things might already work quite well, for example it should not be difficult to discern from (let me use ITERATE's example)

    (iterate (for el in list)
             (finding el minimizing (length el)))
that you're computing argmin using a pure function, which should be easy to do in parallel, whereas the way you'd typically write this in C would be probably difficult for the compiler to confirm as such (it would have to look for pattern such as "if (f(current)<f(best)) { best=current; }" or something like that. (BTW have you seen another loop construct supporting argmin/argmax in another language? I haven't so far.) Whether this could be done somewhat generally and extensibly is something I have to ponder on.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#212

Post by jhgb »

creata wrote: post #154
jhgb wrote: post #150

Earlier quoted context omitted.

They're not "essentially equivalent". A while loop can't begin in one module and end in another. A tail call sequence can. Loops are not modular.

Just to be clear, are you suggesting mutual recursion across modules ?

Just to be clear, I'm using the word "module" here in an extremely generic, language independent way. Basically in the sense of "things that can be compiled individually". In many languages (for example in Lisp), those are functions. A syntactic loop construct ordinarily can't even cross a function boundary.

If your language of choice features formalized modules in the Modula-2 sense, its loops most likely can't span multiple functions even within a single Modula-2-sense module, even if you wanted that (for example, in a state machine with named states as functions which tail-call each other to switch state by transferring control, or something like that).

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#213

Post by t0mas88 »

xvilka wrote: post #104
jstx1 wrote: post #23

Earlier quoted context omitted.

I don't know, x being a potential replacement for y doesn't say all that much. You could have been writing Java or C++ for the past 25 years and got loads of stuff done, solved problems, shipped software, made money etc. Languages are fun to think about but you don't always need to be concerned with every vocal minority of programmers that like to talk about how their language is better than yours. Sometimes that rep…

Choice between Java and native languages (for the backend at least) is obvious due to the performance concerns. Same with Python/Julia.

For Java that's mostly a myth. Modern Java isn't measurably slower than C++ in many cases and faster in some.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#214

Post by fock »

humanrebar wrote: post #201
pilooch wrote: post #182

Loving pytorch but the reasoning remains strange to my ears: Python at all cost, not Julia, because of the ecosystem, well OK. But all bullet points are about things that are easily done right now with libtorch (pytorch underlying C++ core code), and the hassle is... Python. Well rational conclusion would be, just do everything in C++, and bind to Python. Make C++ first citizen here, since in all cases it'll be neede…

I happen to know that pytorch is a pain to maintain in packaging systems. It has a complex build system, many non-python dependencies, and massive build times. I don't know how this factors into the dissatisfaction with a C++ implementation, but I wouldn't be surprised if it were a factor. In other words, python binary wheels are harder to maintain than source-only python packages. And pytorch uses more than a few. I…

I would recommend you never try to compile tensorflow then. Honestly pytorch might take ages and be relatively complex. But 9/10 tries it builds on a recent Fedora, when you get a new release. tensorflow then ... I mean it's not me to judge that, as I don't understand why you would vendor a ton of C-libraries while depending on a single-file v0.0.3 python library...

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#215

Post by StefanKarpinski »

adgjlsfhk1 wrote: post #149
wging wrote: post #144

Earlier quoted context omitted.

> I also think that the JVM object model is quite limiting for numerical computing. They still don’t support value types This is mostly true, but the primitives are value types and you can get some things done with them. (Not enough to make Java good for these use cases, no.) I.e. write float[] instead of Float[] and you have a contiguously allocated region of memory that can be efficiently accessed.

the fact that they couldn't make primitive vs object invisible is just incredibly dumb though. it's one of the many ways c# is purely superior.

C# is certainly more flexible in this regard but I wouldn’t say they made the distinction between primitive and object invisible, they just allowed user-defined “primitives” in the sense that users can define C-style structs that have value semantics. The type system is still bifurcated between value types and reference types. That’s ugly but livable in a static language where you can just disallow mixing the two kinds of types. And it’s not just theoretically ugly: it means you cannot write generic code that works for both kinds of types. Or you can yolo it like C++ templates and just allow whatever by textual substitution—to hell with the semantic differences. But of course then you get all kinds of surprising differences in fundamental behavior based what kinds of types you try to apply your generics to. Which is exactly what happens with C++ templates.

What do you do in a dynamic language where no such separation can be enforced? Or in a static language where you just don’t want that kind of ugly anti-generic bifurcation of your type system? The classic static PL answer is to just make all objects have reference behavior. (Which is the approach Java takes with a pragmatic ugly exception of primitive types.) This is the approach that ML and its derivatives take, but which is why they’re unsuitable for numerical computing—in ML, Haskell, etc. objects have “uniform representation” which means that all objects are represented as pointers (integers are usually represented as special invalid pointers for efficiency), including floating point values. In other words an array of floats is represented in ML et al. as an array of pointers to individual heap-allocated, boxed floats. That makes them incompatible with BLAS, LAPACK, FFTW, and just generally makes float arrays inefficient.

So what does Julia do? Instead of having mutable value and reference types, which have observably different semantics, it has only reference types but allows—and even defaults to—immutable reference types. Why does this help? Because immutable reference types have all the performance and memory benefits of value types! Yet they still have reference-compatible language semantics, because there’s no way to distinguish reference from value semantics without mutation. In other words immutable structs can be references as far as the language semantics are concerned but values as far as the compiler and interop are concerned. And you can even recover efficient mutation whenever the compiler can see that you’re just replacing a value with a slightly modified copy—and compilers are great at that kind of optimization. So all you give up is the ability to mutate your value types—which you don’t even want to allow for must numeric types anyway—and which you can simulate by replacement with modification. This seems like a really good trade off and it’s a little surprising that more languages don’t make it.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#216

Post by pm90 »

xvilka wrote: post #104
jstx1 wrote: post #23

Earlier quoted context omitted.

I don't know, x being a potential replacement for y doesn't say all that much. You could have been writing Java or C++ for the past 25 years and got loads of stuff done, solved problems, shipped software, made money etc. Languages are fun to think about but you don't always need to be concerned with every vocal minority of programmers that like to talk about how their language is better than yours. Sometimes that rep…

Choice between Java and native languages (for the backend at least) is obvious due to the performance concerns. Same with Python/Julia.

This might have been the case in the past, but the JVM is now good enough that you get very close to native (and sometimes better) performance with Java.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#217

Post by pm90 »

ChrisRackauckas wrote: post #206
freemint wrote: post #199

Earlier quoted context omitted.

In certain common lisp deployment you might do that. There are legends from Common Lisp users where this is very much done. > When one of the customer support people came to me with a report of a bug in the editor, I would load the code into the Lisp interpreter and log into the user's account. If I was able to reproduce the bug I'd get an actual break loop, telling me exactly what was going wrong. Often I could fix…

Call me a cynic but I would hope almost no devs had the power to live update the production codebase. That sounds crazy to me. To me, that kind of thing should only be done with (approved/verified) versions which are cut and tested before deployment, and deployment should act like an unchangeable immutable compiled binary until the next binary is ready. Not only is this legally required in many scientific cases, but…

I understand where you’re coming from, it’s the production robustness paradigm of someone who has worked with such systems and knows what kind of controls are required to release code. eg live editing like that would almost certainly go against SOC2 controls.

However, I’m also tempted very much by a system that allows for fixing things so quickly, because it provides for a delightful customer experience. Most customers don’t report most issues, only when something is persistently wrong or is something they can’t get around. So being able to quickly fix it this way seems genuinely amazing. If I were to take this idea further… maybe some sort of customer isolated canary deploy/ feature flag might be the way to express this customer experience in todays world.

The problem is always that you might break other things with your fix. Maybe a battery of testing before canary deploys can raise confidence about the validity of the fix.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#218

Post by dunefox »

nothrowaways wrote: post #202

Why not go? Go beats Julia in parts where python is not good at. Is it because fb vs Google?

It's because Go is a 90s (procedural) language in 2021 and I would maybe use it in a parallel universe where many other languages don't exist.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#219

Post by adgjlsfhk1 »

humanrebar wrote: post #201
pilooch wrote: post #182

Loving pytorch but the reasoning remains strange to my ears: Python at all cost, not Julia, because of the ecosystem, well OK. But all bullet points are about things that are easily done right now with libtorch (pytorch underlying C++ core code), and the hassle is... Python. Well rational conclusion would be, just do everything in C++, and bind to Python. Make C++ first citizen here, since in all cases it'll be neede…

I happen to know that pytorch is a pain to maintain in packaging systems. It has a complex build system, many non-python dependencies, and massive build times. I don't know how this factors into the dissatisfaction with a C++ implementation, but I wouldn't be surprised if it were a factor. In other words, python binary wheels are harder to maintain than source-only python packages. And pytorch uses more than a few. I…

Julia actually makes this problem way better for 2 different reasons. the first is that you don't need nearly as much C/Fortran when you are working in a fast language. the second is that Julia has binarybuilder which is a really good system for delivering reproduceable binaries that can be distributed. To show how well this works, try adding the Cuda Julia library. it just works without any of the issues python has.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#220

Post by tfehring »

driscoll42 wrote: post #100
FridgeSeal wrote: post #92

This seems so silly to me. It’s PyTorch-if they said “the next version of PyTorch will be in Julia, the ecosystem would shift accordingly. They’re practically saying “this language has every feature we need and want, most of them already existing, but we’re going to continue re-inventing them in this objectively less suitable language because we clearly wish to make life harder for ourselves”

Or I read it as "We want to make life as easy for our userbase as possible, so we will put more work on ourselves to make our users lives easier" which is an attitude I very much appreciate.

Does keeping as much of the codebase as possible in Python (or keeping the fast parts in C++) actually make things easier for the userbase, or do they just care about having a first-class interface in Python regardless of the implementation language?
Post reply on HN