High-Performance GPU Computing in the Julia Programming Language (2017) • Hacker News Board

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#21

Post by mirekrusin » Mon, Oct 28, 2019, 3:56 AM UTC

zzleeper wrote: post #15

siproprio wrote: post #5

Earlier quoted context omitted.

In theory Julia is supposed to be fantastic. In practice, things either don't exist, or are poorly implemented: Plotting simple things take 30 seconds. And that's if you don't count the time it takes to `] add Plots`, especially on Windows! And the REPL is broken. And the editor is slow and annoying (Juno or vscode). And documentation ranges from poor (no examples, buggy between platforms, broken links due to version…

Similar experience here. Did a comparison of a bunch of statistical tools (R, Matlab, Julia, Python, etc.) on small-ish datasets. Used the latest versions in all cases, in Windows 10. All but Julia ran the regressions in Sure, the usual answer is "well its an initial cost, its faster after that" but not all my code would otherwise take days to run. As long as "using CSV" takes 10 seconds, I'm out.

Low latency development flow is slightly different in Julia. You should startup you process and reload updated code with revise. You won't have this problem. Obviously compilation performance improvements will be very welcome when they arrive but it's not a deal breaker because of this revise based flow.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#22

Post by jlebar » Mon, Oct 28, 2019, 4:29 AM UTC

sytelus wrote: post #8

Why this infrastructure is so tightly coupled with CUDA? CUDA is very specific and closed APIs for NVidia hardware only. Programming languages should focus on more general primitives that might work on NVidia or TPUs or something else. PyTorch also has CUDA all over in its APIs and its frustrating to see such tight binding with closed one company API. Also take a look at OpenCL.

> Why this infrastructure is so tightly coupled with CUDA?

It's not. It uses LLVM, which can easily target AMD GPUs. (Whether the Julia folks have invested in making this work, I dunno, but it's not Extremely Hard.)

Understandably nvidia gives you the wrong impression.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#23

Post by shaklee3 » Mon, Oct 28, 2019, 4:32 AM UTC

sytelus wrote: post #14

dlphn___xyz wrote: post #10

Earlier quoted context omitted.

there are specific benefits of cuda over opencl: see https://arxiv.org/vc/arxiv/papers/1005/1005.2581v1.pdf

Yes, but is cuda going to keep its edge 10 years down the line? Do I want to hardcode my algorithms so tightly with today's cuda APIs? Can there be better more generic primitives that are agnostic of propitiatory cuda APIs but would support it as backend without too much perf hit?

AMD has a search and replace library that's API compatible with many cuda functions now. It hasn't caught on yet, but if they release decent hardware soon, it might.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#24

Post by [deleted] » Mon, Oct 28, 2019, 4:36 AM UTC

shmerl wrote: post #20

darknoon wrote: post #9

Earlier quoted context omitted.

It's because CUDA performs better. It's not nice, but it's the situation we're living in. Particularly AMD support and performance are lot.

It doesn't perform better than what you can do in Vulkan. It's simply more entrenched.

[deleted]

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#25

Post by rrss » Mon, Oct 28, 2019, 4:42 AM UTC

shmerl wrote: post #18

> The performance possibilities of GPUs can be democratized by providing more high-level tools that are easy to use by a large community of applied mathematicians and machine learning programmers. How exactly CUDA is "democratizing" anything, if it's tied to Nvidia? Vulkan backend would make more sense for that purpose.

That sentence explains perfectly well what it means by democratizing, and how is independent of the platform being tied to nvidia.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#26

Post by cbkeller » Mon, Oct 28, 2019, 4:55 AM UTC

siproprio wrote: post #5

adamnemecek wrote: post #4

Julia is one of my fav languages. For numerical computing, neither python + numpy, nor matlab come even close. The interop is nuts. To call, say numpy fft, you just do using PyCall np = pyimport("numpy") res = np.fft.fft(rand(ComplexF64, 10)) No casting back and forth. This is a toy example, julia ofc has fftw bindings. Interop with C++, MATLAB, Mathematica etc is similarly simple.

In theory Julia is supposed to be fantastic. In practice, things either don't exist, or are poorly implemented: Plotting simple things take 30 seconds. And that's if you don't count the time it takes to `] add Plots`, especially on Windows! And the REPL is broken. And the editor is slow and annoying (Juno or vscode). And documentation ranges from poor (no examples, buggy between platforms, broken links due to version…

Things are still improving, but I already find it quite usable in practice. Yes, precompilation can take a while, but once you start using the language regularly you hardly notice it, since you only need to re-precompile after installing updates -- which ends up being a small proportion of the time. Same with "time to first plot", since I almost always have a session already running already these days.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#27

Post by backdoorsgalore » Mon, Oct 28, 2019, 5:01 AM UTC

shmerl wrote: post #20

darknoon wrote: post #9

Earlier quoted context omitted.

It's because CUDA performs better. It's not nice, but it's the situation we're living in. Particularly AMD support and performance are lot.

It doesn't perform better than what you can do in Vulkan. It's simply more entrenched.

WebGPU version of TensorFlow is in the works. Hopefully, that will dethrone NVIDIA. AFAIK, it's significantly more performant than WebGL, and it works on top of Vulkan, D3D and Metal. The heck with NVIDIA. Such a bunch of control freaks.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#28

Post by shmerl » Mon, Oct 28, 2019, 5:05 AM UTC

rrss wrote: post #25

shmerl wrote: post #18

> The performance possibilities of GPUs can be democratized by providing more high-level tools that are easy to use by a large community of applied mathematicians and machine learning programmers. How exactly CUDA is "democratizing" anything, if it's tied to Nvidia? Vulkan backend would make more sense for that purpose.

That sentence explains perfectly well what it means by democratizing, and how is independent of the platform being tied to nvidia.

Can you elaborate please? I was under the impression that CUDA is tied to Nvidia, unless you mean there are now working shims for other GPUs.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#29

Post by refresh-creds » Mon, Oct 28, 2019, 5:37 AM UTC

sytelus wrote: post #14

dlphn___xyz wrote: post #10

Earlier quoted context omitted.

there are specific benefits of cuda over opencl: see https://arxiv.org/vc/arxiv/papers/1005/1005.2581v1.pdf

Yes, but is cuda going to keep its edge 10 years down the line? Do I want to hardcode my algorithms so tightly with today's cuda APIs? Can there be better more generic primitives that are agnostic of propitiatory cuda APIs but would support it as backend without too much perf hit?

That paper is already about 10 years old so I think you are being trolled.

Re: High-Performance GPU Computing in the Julia Programming Language (2017)

#30

Post by Athas » Mon, Oct 28, 2019, 5:43 AM UTC

darknoon wrote: post #9

sytelus wrote: post #8

Why this infrastructure is so tightly coupled with CUDA? CUDA is very specific and closed APIs for NVidia hardware only. Programming languages should focus on more general primitives that might work on NVidia or TPUs or something else. PyTorch also has CUDA all over in its APIs and its frustrating to see such tight binding with closed one company API. Also take a look at OpenCL.

It's because CUDA performs better. It's not nice, but it's the situation we're living in. Particularly AMD support and performance are lot.

Are you certain that the story is as simple as "CUDA performs better"? It's common folklore, but I have seen little evidence. The only situations I know of when CUDA performs better is when CUDA-specific features are used (if they are relevant for whatever problem is at hand). Also, CUDA libraries (like cuBLAS or cuFFT) tend to be more efficient than their OpenCL equivalent, which is likely because much more work has gone into them. I have also noted that the CUDA compiler is willing to use less accurate (but faster) floating-point instructions by default (for things like e.g. inverse square root), where you need to pass options to the OpenCL compiler for it to do the same. This will matter for some programs.

In fact, I have run tens of thousands of lines of essentially equivalent CUDA and OpenCL code (automatically generated) on the same hardware, and performance was in all cases very similar[0]. If anything, CUDA was actually slower than average (but in the cases I investigated, this was down to arbitrary differences like the CUDA compiler not unrolling some loops as aggressively and such).

[0]: https://futhark-lang.org/blog/2019-02-08-futhark-0.9.1-relea...