Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

dev-discuss.pytorch.org

Post reply on HN
Print view View original

251–260 of 291 posts

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#251

Post by bjourne »

ddragon wrote: post #242
bjourne wrote: post #236

Earlier quoted context omitted.

According to this discussion thread, Julia requires the number of kernel threads to be specified at startup: https://discourse.julialang.org/t/does-multithreading-requir... This seem to be more like pyprocessing and much more limited than the pthreads interface.

I didn't go deep on Julia's multithreading, but what he is saying is that Julia uses an MxN threading (I think nowadays if you don't specify it at startup it will just use one for each cpu thread), which is the same as a language I did most of distributed programming (elixir/erlang), and as far as I know it's the same as Go. Having 1 kernel thread for each CPU thread means that your program can use all available CPU…

But that scheme is the same as green threading and has the same faults. Start Julia with one system thread. Run one infinite loop in one green thread and another in another green thread. One of the loops will run while the other will wait until the other completes (which it never does). This is inferior to Java and C# which both uses system threads by default, allowing both infinite loops to run. Erlang/Elixir has the same problem. You can run thousands of green threads, but if one of them is stupid enough to call a blocking C function then all others have to wait.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#252

Post by upbeat_general »

edgyquant wrote: post #45
geofft wrote: post #15

> Julia says: > A language must compile to efficient code, and we will add restrictions to the language (type stability) to make sure this is possible. > A language must allow post facto extensibility (multiple dispatch), and we will organize the ecosystem around JIT compilation to make this possible. > The combination of these two features gives you a system that has dynamic language level flexibility (because you h…

The main reason is that a lot of scientific computing depends on ancient libraries written in Fortran or C, and a lot of newer libraries depend on GPUs for their matrix operations so need to be able to make calls to C/C++ modules that run directly to the GPU. This probably isn’t impossible for Java but that was never meant to be it’s niche, it doesn’t compile to machine instructions and prefers portability.

I know FFI in Java is certainly possible and I’ve seen it at least a few times personally; is it that much harder than in Python or just less common due to the need?

Python doesn’t compile to machine instructions either and there’s nothing that prevents GPU access from Java. In fact I’d bet in many cases pure Java beats Python + C library though it obviously depends on how much has to be written in pure python.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#253

Post by ddragon »

bjourne wrote: post #251
ddragon wrote: post #242

Earlier quoted context omitted.

I didn't go deep on Julia's multithreading, but what he is saying is that Julia uses an MxN threading (I think nowadays if you don't specify it at startup it will just use one for each cpu thread), which is the same as a language I did most of distributed programming (elixir/erlang), and as far as I know it's the same as Go. Having 1 kernel thread for each CPU thread means that your program can use all available CPU…

But that scheme is the same as green threading and has the same faults. Start Julia with one system thread. Run one infinite loop in one green thread and another in another green thread. One of the loops will run while the other will wait until the other completes (which it never does). This is inferior to Java and C# which both uses system threads by default, allowing both infinite loops to run. Erlang/Elixir has th…

That issue actually doesn't happen with regular elixir, since it's immutable and stateless, the scheduler doesn't wait for the process to voluntarily wield (it's always safe to switch, so it just gives some fixed time for each process, which makes it very low latency and very reliable but not very efficient at any individual task due to all the switching). Calling other languages from erlang/elixir create dirty processes that can't be scheduled this way and may cause those issues.

Since I didn't use dirty processes in elixir it did make me forget about this obvious issue you pointed out, that for a mutable language like Julia can happen in every thread, but that's not something that limits the expressiveness of the model, but something that requires consideration to avoid while programming and language level mechanisms to protect the thread (at the very least the ability to define timeouts that can throw an exception on any spawned process) or maybe a future framework on top of it that handles this in a safer way (something like Akka). I can only hope Julia can achieve the full potential of it's multithreading model.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#255

Post by adsharma »

jszymborski wrote: post #21

I've been thinking a lot about trying to use functional dialect of Python like Coconut[0] and Hy[1] w/ JAX so I can write functional DL code. Glad to see functorch[3] as PyTorch is the library I have the most experience with. [0] http://coconut-lang.org/ [1] https://docs.hylang.org/en/alpha/ [2] https://github.com/google/jax [3] https://github.com/pytorch/functorch

Also see:

https://github.com/adsharma/py2many/blob/main/doc/langspec.m...

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#257

Post by eigenspace »

bjourne wrote: post #251
ddragon wrote: post #242

Earlier quoted context omitted.

I didn't go deep on Julia's multithreading, but what he is saying is that Julia uses an MxN threading (I think nowadays if you don't specify it at startup it will just use one for each cpu thread), which is the same as a language I did most of distributed programming (elixir/erlang), and as far as I know it's the same as Go. Having 1 kernel thread for each CPU thread means that your program can use all available CPU…

But that scheme is the same as green threading and has the same faults. Start Julia with one system thread. Run one infinite loop in one green thread and another in another green thread. One of the loops will run while the other will wait until the other completes (which it never does). This is inferior to Java and C# which both uses system threads by default, allowing both infinite loops to run. Erlang/Elixir has th…

> Start Julia with one system thread. Run one infinite loop in one green thread and another in another green thread. One of the loops will run while the other will wait until the other completes (which it never does). This is inferior

If you want julia to use multiple system threads, why are you suggesting one not use system threads for this test? All you have to do is start julia with multiple threads and it'll use those threads for your infinite loops.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#258

Post by xvilka »

t0mas88 wrote: post #213
xvilka wrote: post #104

Earlier quoted context omitted.

Choice between Java and native languages (for the backend at least) is obvious due to the performance concerns. Same with Python/Julia.

For Java that's mostly a myth. Modern Java isn't measurably slower than C++ in many cases and faster in some.

Nevertheless, every time I run JetBrains IDEs it works much slower than QtCreator, for example, and eats way too much memory compared to the native equivalents. Hardly a myth if I can see it with my own eyes. Same with Ghidra vs Binary Ninja or IDA Pro. And so on.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#259

Post by Daishiman »

nextos wrote: post #128
driscoll42 wrote: post #100

Earlier quoted context omitted.

Or I read it as "We want to make life as easy for our userbase as possible, so we will put more work on ourselves to make our users lives easier" which is an attitude I very much appreciate.

In the long run I think moving to Julia would make a lot of sense. I have used MATLAB, R, Python and Julia extensively for doing all sorts of data related things during the last 20 years. Julia is incredibly easy to work with, very elegant and really efficient. R and Python have always felt clumsy in some ways, and hard to write really performant code, even if you are more proficient in Python! As a seasoned Lisper a…

But Julia doesn't have a good story for anything else besides that.

You can take a Python web server process, have a request call a task that uses NumPy and OpenCV and scikit-learn, get that back, and you're done, all in the same language.

Julia's community does not seem to have aspirations beyond high-performance math code, which is great for its use, but I'm not going to learn Julia just for that when I can implement the entirety of a development pipeline in Python and have all the other niceties that come with it.

Re: PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly)

#260

Post by adgjlsfhk1 »

tfehring wrote: post #222
chaxor wrote: post #158

Earlier quoted context omitted.

The way I see Julia now is how I saw python ~12 years ago

I agree, but it's probably closer to 20 years IMO, which makes sense given that Python has been around for 21 years longer. By 2009 Python was already in widespread use at Google/YouTube and (I think) Mozilla, plus startups like Reddit and Dropbox. Hell, Steve Huffman wrote "Lots of people have written web applications in Python, and there's plenty of code from which to learn" as one justification for Reddit's Python…

Julia has a decent amount of penetration in the private sector, just not in "tech". If you look at JuliaComputing's clients, you'll see a lot of private customers, they're just not the type that typically talks a lot about the tech they're using.
Post reply on HN