Why programming languages matter [video]

Very little innovation in programming languages has happened regarding new realities at the hardware level especially transition from serial to parallel execution.

Re: Why programming languages matter [video]

#5

Post by anon291 » Fri, Oct 27, 2023, 9:59 PM UTC

jokoon wrote: post #3

oddly, there are very few job positions to work in things related to programming languages.

I feel like every major company is hiring for AI compiler engineers right now (based on my inbox at least). May not be directly related to general 'programming languages', but my take, as someone in the industry, is that all the PL people are working on this right now.

Re: Why programming languages matter [video]

#6

Post by nomel » Fri, Oct 27, 2023, 10:14 PM UTC

mjfl wrote: post #4

Very little innovation in programming languages has happened regarding new realities at the hardware level especially transition from serial to parallel execution.

Does anyone have any good counterpoints to this? From my, naive, perspective, this seems to be relatively true.

I've always assumed that, by now, I would be able to write code, in a semi mainstream language, and it would be made somewhat parallel, by the compiler. No need for threads, or me thinking of it.

There's projects like https://polly.llvm.org, but I guess I assumed there would be more progress through the decades.

Re: Why programming languages matter [video]

#7

Post by tylerhou » Fri, Oct 27, 2023, 10:16 PM UTC

nomel wrote: post #6

mjfl wrote: post #4

Very little innovation in programming languages has happened regarding new realities at the hardware level especially transition from serial to parallel execution.

Does anyone have any good counterpoints to this? From my, naive, perspective, this seems to be relatively true. I've always assumed that, by now, I would be able to write code, in a semi mainstream language, and it would be made somewhat parallel, by the compiler. No need for threads, or me thinking of it. There's projects like https://polly.llvm.org , but I guess I assumed there would be more progress through the de…

Take a look at Halide, which can autovectorize and multi-thread graphics computations (but does require a restricted language).

Re: Why programming languages matter [video]

#8

Post by nomel » Fri, Oct 27, 2023, 10:18 PM UTC

tylerhou wrote: post #7

nomel wrote: post #6

Earlier quoted context omitted.

Does anyone have any good counterpoints to this? From my, naive, perspective, this seems to be relatively true. I've always assumed that, by now, I would be able to write code, in a semi mainstream language, and it would be made somewhat parallel, by the compiler. No need for threads, or me thinking of it. There's projects like https://polly.llvm.org , but I guess I assumed there would be more progress through the de…

Take a look at Halide, which can autovectorize and multi-thread graphics computations (but does require a restricted language).

I was thinking outside of "embarrassingly parallel" [1] type work. :) But, that is fair.

[1] https://en.wikipedia.org/wiki/Embarrassingly_parallel#:~:tex....

Re: Why programming languages matter [video]

#9

Post by 7734128 » Fri, Oct 27, 2023, 10:36 PM UTC

nomel wrote: post #6

mjfl wrote: post #4

Very little innovation in programming languages has happened regarding new realities at the hardware level especially transition from serial to parallel execution.

Does anyone have any good counterpoints to this? From my, naive, perspective, this seems to be relatively true. I've always assumed that, by now, I would be able to write code, in a semi mainstream language, and it would be made somewhat parallel, by the compiler. No need for threads, or me thinking of it. There's projects like https://polly.llvm.org , but I guess I assumed there would be more progress through the de…

Especially when doing things like mapping or list comprehension. I'd love to be able to do operations on collections in parallel by simply marking my functions as pure. Just a simple xs.map {x -> f(x)} combined with "f" marked as pure confirmed by the compiler to make the magic happen.

Re: Why programming languages matter [video]

#10

Post by jcranmer » Fri, Oct 27, 2023, 10:39 PM UTC

nomel wrote: post #6

mjfl wrote: post #4

Very little innovation in programming languages has happened regarding new realities at the hardware level especially transition from serial to parallel execution.

Does anyone have any good counterpoints to this? From my, naive, perspective, this seems to be relatively true. I've always assumed that, by now, I would be able to write code, in a semi mainstream language, and it would be made somewhat parallel, by the compiler. No need for threads, or me thinking of it. There's projects like https://polly.llvm.org , but I guess I assumed there would be more progress through the de…

Proving legality of transformations in the compiler is frequently impossible. Consequently, the main mode of implementation has been to essentially think of the problems in terms of the user saying that this loop is parallel, please make it run in parallel. OpenMP or Rust's rayon crate, for example. The other similar innovation has been programming SIMD as if each lane were an independent thread, which is essentially the model of ispc or CUDA (or #pragma omp simd, natch).

The other big impossible task is that most code isn't written to be able to take advantage of theoretical autoparallelization--you really want data to be in struct-of-arrays format, but most code tends to be written in array-of-struct format. This means that vectorization cost model (even if proven, whether by user assertion or sufficiently smart compiler, legal) sees it needs to do a lot of gathers and scatters and gives up on a viable path to vectorization really quickly.