Live data from Hacker News

I want a good parallel language [video]

youtube.com

51–60 of 69 posts

Re: I want a good parallel language [video]

#51
post #5

There were a few languages designed specifically for parallel computing spurred by DARPA's High Productivity Computing Systems project. While Fortress is dead, Chapel is still being developed.

Those languages were not effective in practice. The kind of loop parallelism that most people focus on is the least interesting and effective kind outside of niche domains. The value was low. Hardware architectures like Tera MTA were much more capable but almost no one could write effective code for them even though the language was vanilla C++ with a couple extra features. Then we learned how to write similar softwa…

That's why programming languages are important for solving this problem.

The syntax and semantics should constrain the kinds of programs that are easy to write in the language to ones that the compiler can figure out how to run in parallel correctly and efficiently.

That's how you end up with something like Erlang or Elixir.

Re: I want a good parallel language [video]

#52
post #4

SQL. It is a joke, but an SQL engine can be massively parallel. You just don't know it, it just gives you what you want. And in many ways the operations resembles what you do for example in CUDA. CUDA backend for DuckDB or Trino would be one of my go-to projects if i was laid off.

My issue with SQL is lack of composability and difficulty of debugging intermediate results.

You can use SQL CTE's and/or VIEW's as a composable abstraction over queries and inspect intermediate results. The language features are there.

Re: I want a good parallel language [video]

#53
post #5

There were a few languages designed specifically for parallel computing spurred by DARPA's High Productivity Computing Systems project. While Fortress is dead, Chapel is still being developed.

Those languages were not effective in practice. The kind of loop parallelism that most people focus on is the least interesting and effective kind outside of niche domains. The value was low. Hardware architectures like Tera MTA were much more capable but almost no one could write effective code for them even though the language was vanilla C++ with a couple extra features. Then we learned how to write similar softwa…

This was, I think, the greatest strength of MapReduce. If you could write a basic program you could understand the map, combine, shuffle and reduce operations. MR and Hadoop etc. would take care of recovering from operational failures like disk or network outages by idempotencies in the workings behind the scenes, and programmers could focus on how data was being transformed, joined, serialized, etc.

To your point, we also didn't need a new language to adopt this paradigm. A library and a running system were enough (though, semantically, it did offer unique language-like characteristics).

Sure, it's a bit antiquated now that we have more sophisticated iterations for the subdomains it was most commonly used for, but it hit a kind of sweet spot between parallelism utility and complexity of knowledge or reasoning required of its users.

Re: I want a good parallel language [video]

#54

ctrl-f Erlang Nothing yet? Damn...

Erlang operates on a higher level, with much, much larger chunks. Any sensible modern system will, for instance, have some sort of execution context (thread/green thread/continuation/task/whatever) associated with a single incoming HTTP request. That's very nice, and not going anywhere.

However Erlang has very little to say about parallelization of loops, or in the levels between a single loop and a HTTP request.

Nor would it be a good base for such things; if you're worried about getting maximum parallel performance out of your CPUs you pretty much by necessity need to start from a base where single-threaded performance is already roughly optimal, such as with C, C++, or Rust. Go at the very outside, and that's already a bit of a stretch in my opinion. BEAM does not have that level of single-threaded performance. There's no point in making what BEAM does fully utilize 8 CPUs in this sort of parallel performance when all that does is get you back to where a single thread of Rust can run.

(I think this is an underappreciated aspect of trying to speed things up with multiple CPUs. There's no point straining to get 8 CPUs running in some sort of complicated perfect synchronization in your slow-ish language when you could just write the same thing in a compiled language and get it on one CPU. I particularly look at the people who think that GIL removal in Python is a big deal for performance and wonder what they're thinking... a 32-core machine parallelizing Python code perfectly, with no overhead, might still be outperformed by a single-core Go process and would almost certainly be beated by a single-core Rust process. And perfect parallelization across 32 cores is a pipe dream. Unless you've already maxed out single-core performance, you don't need complicated parallelization, you need to write in a faster language to start with.)

Re: I want a good parallel language [video]

#55

ctrl-f Erlang Nothing yet? Damn...

Yeah, i too was looking for Erlang. The thing i would really like to see is some research on how to run the Erlang concurrency model on a GPU.

There's no need for research. The answer is simple: You can't run Erlang concurrency on a GPU. GPUs fundamentally get their advantage by running the same operations on a huge set of cores across different data. They aren't just Platonically faster than CPUs, they're faster than CPUs on very, very specific tasks. Out of the context of those tasks, they are in fact massively, massively slower.

Some of the operations Erlang does, GPUs don't even want to do at all, including basic things like pattern matching. GPUs do not want that sort of code at all.

"Erlang" is being over specific here. No conventional CPU language makes sense on a GPU at all.

Re: I want a good parallel language [video]

#56
post #4

SQL. It is a joke, but an SQL engine can be massively parallel. You just don't know it, it just gives you what you want. And in many ways the operations resembles what you do for example in CUDA. CUDA backend for DuckDB or Trino would be one of my go-to projects if i was laid off.

My issue with SQL is lack of composability and difficulty of debugging intermediate results.

Check out https://prql-lang.org/

It solves all the warts of sql while still being true to its declarative execution. Trailing commas, from statement first and reads as a a composable pipeline, temporary variables for expressions, intuitive grouping.

Re: I want a good parallel language [video]

#58

ctrl-f Erlang Nothing yet? Damn...

Yeah, i too was looking for Erlang. The thing i would really like to see is some research on how to run the Erlang concurrency model on a GPU.

Something like vine lang or something built on interaction nets might be close to what you are looking for. It can run on GPU.

Re: I want a good parallel language [video]

#60
post #55

Earlier quoted context omitted.

Yeah, i too was looking for Erlang. The thing i would really like to see is some research on how to run the Erlang concurrency model on a GPU.

There's no need for research. The answer is simple: You can't run Erlang concurrency on a GPU. GPUs fundamentally get their advantage by running the same operations on a huge set of cores across different data. They aren't just Platonically faster than CPUs, they're faster than CPUs on very, very specific tasks. Out of the context of those tasks, they are in fact massively, massively slower . Some of the operations E…

Not what i meant (these are superficialities).

Erlang is a concurrency-oriented language though its concurrency architecture (multicore/node/cluster/etc.) is different from that modeled by GPUs (Vectorized/SIMD/SIMT/etc.) Since share-nothing Processes (so-called Actor model) are at the heart of the Erlang Run Time System(ERTS)/BEAM it is easy to imagine a "group of Erlang processes" being mapped directly to a "group of threads in a warp on a GPU". Of course the Erlang scheduler being different (it is reduction based and not time sliced) one would need to rethink some fundamental design decisions but that should not be too out-of-the-way since the system as a whole is built for concurrency support. The other problem would be memory transfers between CPU and GPU (while still preserving immutability) but this is a more general one.

You can call out to CUDA/OpenCL/etc. from Erlang through its C interface (Kevin Smith did a presentation years ago) but i have seen no new research since then. However, there has been some new things in Elixir land notably "Nx" (Numerical Elixir) and "GPotion" (a DSL for GPU programming in Elixir).

But note that none of the above is aimed at modifying the Erlang language/runtime concurrency model itself to map to GPU models which is what i would very much like to see.

Post reply on HN