Live data from Hacker News

The free lunch is over: a fundamental turn toward concurrency in software (2005)

gotw.ca

51–60 of 101 posts

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#51
post #38

Earlier quoted context omitted.

> If only GPUs were more common They are very common. In 2021 on Windows, is safe to assume the GPU does at least D3D 11.0. The last GPU which does not is Intel Sandy Bridge, discontinued in 2013. D3D 11.0 supports lots of features about GPGPU. The main issue with D3D11 is FP64 support being optional, the hardware may or may not support. > and easier to work with I don’t find neither DirectCompute 11, nor CUDA, excep…

> I don’t find neither DirectCompute 11, nor CUDA, exceptionally hard to work with. It still sucks that, if you're not careful or unlucky, GPUs may interfere with normal video operation during setup. Also on one machine I'm getting "GPU lost" error messages every once in a while during a long computation, and I have no other option than to reboot my machine. Further, the form factor sucks. Every card comes with 4 or…

> Further, the form factor sucks. Every card comes with 4 or more DVI connectors which I don't need.

DVI connectors are huge! I've never seen a card with more than 2 of them.

You working with something more exotic than a PCI-e card?

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#52
post #47
post #37

I find it interesting that the language that best deals with parallelism (IMO) was invented significantly before we moved in the multi-core direction. The programming language landscape evolved and developed in the face of multi-code - async being a classic example. But the language that's often held up as the best solution to any given parallelism problem is Erlang. Erlang was built as a good programming model for c…

The writing was on the wall as early as the beginning of the Pentium 4 era. There was this assumption that clock speed would just continue to grow at the insane rates it did in the 80s and 90s, and then reality caught up when the Netburst architecture didn't scale well enough to avoid heat issues and crazy insane pipelines. When Intel rolled out the Core architecture, they went back to a P6-derived design, a microarc…

Erlang was begun in 1986. It left the lab and was used for production products first in 1996. The Pentium 4 came out in 2000.

So, an interesting aside, but not sure it's relevant to the parent comment.

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#53
post #47
post #37

I find it interesting that the language that best deals with parallelism (IMO) was invented significantly before we moved in the multi-core direction. The programming language landscape evolved and developed in the face of multi-code - async being a classic example. But the language that's often held up as the best solution to any given parallelism problem is Erlang. Erlang was built as a good programming model for c…

The writing was on the wall as early as the beginning of the Pentium 4 era. There was this assumption that clock speed would just continue to grow at the insane rates it did in the 80s and 90s, and then reality caught up when the Netburst architecture didn't scale well enough to avoid heat issues and crazy insane pipelines. When Intel rolled out the Core architecture, they went back to a P6-derived design, a microarc…

And initial forays into many cores was decades earlier then that. Aside from big iron, going back to the early 60s, there was some business class devices like those made by Sequent:

https://en.m.wikipedia.org/wiki/Sequent_Computer_Systems

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#54
post #37

I find it interesting that the language that best deals with parallelism (IMO) was invented significantly before we moved in the multi-core direction. The programming language landscape evolved and developed in the face of multi-code - async being a classic example. But the language that's often held up as the best solution to any given parallelism problem is Erlang. Erlang was built as a good programming model for c…

> no programs needed changing at all to get the full advantage of all the cores (barring some special cases)

Some programs just don't have any available parallelism in them - no matter how many processes you split them up into. You need to build parallel algorithms and data structures. That's still often a research-level topic.

Erlang's not going to magic parallelism out of thin air in a program that doesn't have any.

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#55
post #29
post #28

Earlier quoted context omitted.

What is Python missing in your eyes? Cooperative multitasking is a relatively recent thing to be in vogue again, and python has had good support for multithreading and multiprocessing since forever.

Without in-process multi-threading with OS threads, writing performant parallel code is nearly impossible. Most of the time you have to drop down to C/C++, disable the GIL and implement parallelism there.

Every time someone brings up the GIL and performance, the answer is pretty much always "use a different language for the places you need especially high performance". All this sturm and drang about Python not being performant enough is missing the point. It's not supposed to be the answer to special cases where maximum performance is required! It's a general-purpose language that is built around community and easy readability and elimination of boilerplate, and be good-enough for a vast majority of usecases. Just like any language, there are usecases where it is not appropriate. Clearly, it has succeeded given its significant reach and longevity.

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#56
I agree with this article a lot.

The layers over layers of software bloat increased with the hardware speed. The general trade-off is: We make the software as slow as possible to have the shortest development time with the least educated programmers. "Hey web dev intern, click me a new UI in the next two hours! Hurry!"

This intern-clicks-a-new-UI works because we have a dozen layers of incomprehension-supporting-technologies. You don't need to know how most parts of the machine are working, Several libraries, VMs, and frameworks will make you a bed of roses.

My point is that we a overdoing it with the convenience for the developers. Today there is way too much complexity and bloat in our systems. And there are not enough programmers trained to handle memory management and similar low-level tasks. Or their bosses wouldn't allow it because the deadline, you know.

I think the general trend is bad because there is no free lunch. No silver bullet. Everything is a trade-off. For example C is still important because C's trade-off between programming convenience and runtime efficiency is very good. You pay a lot and you get a lot.

This is also true for parallel programming. To write highly efficient parallel code you need skill and education. No silver bullet tooling will make this less "hard".

And here I see the irony. Faster CPUs were used to have lower educated devs delivering quicker. More parallel CPUs need higher skilled devs working slower to utilize the chip's full potential.

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#57
post #37

I find it interesting that the language that best deals with parallelism (IMO) was invented significantly before we moved in the multi-core direction. The programming language landscape evolved and developed in the face of multi-code - async being a classic example. But the language that's often held up as the best solution to any given parallelism problem is Erlang. Erlang was built as a good programming model for c…

> no programs needed changing at all to get the full advantage of all the cores (barring some special cases) Some programs just don't have any available parallelism in them - no matter how many processes you split them up into. You need to build parallel algorithms and data structures. That's still often a research-level topic. Erlang's not going to magic parallelism out of thin air in a program that doesn't have any…

Kind of strawmanning here, no? Obviously sequential tasks are run sequentially. The OP isn't saying "magic" happens. Just, Erlang was written from the beginning to encourage parallelized code. Its whole programming model is around that. Unlike pretty much every language it was contemporary with, which assumed that the default was a single unit of execution (and that code running in parallel was the exception).

Think of task scheduling; the normal Algol/C family of languages would say "create a priority queue for your tasks, wait until the top of the queue is ready to run, pull it off, run it, repeat". Very sequential thinking. Of course, you then end up with issues around what happens if the processes take non-negligible amounts of time, what happens if you need to add new items into the queue, etc. At the lowest level that might be solved with interrupts, at a higher level it might be threads or separate processes, but you still have to worry about concurrent data access (locks and the like around the queue). But both the general approach, and the 'best practice' guidelines, especially prior to multi-core, (due to the difficulties of shared data), were/are to minimize concurrency.

Erlang, you just spin up every task as its own process. The process sleeps until it's time to do something, then it does it. The language encourages you to solve the problem in a parallel manner, and it did so from the beginning.

So, yes, you "need to build parallel algorithms and data structures". The point is, Erlang took advantage of them, as a language, even before the hardware could. The way you would write Erlang for a single core was the way you would write Erlang for many cores, which was unique for a language at the time, and the OP is arguing that not only was Erlang unique in doing that, but that there really isn't a better model for handling parallelism from a language and correctness standpoint (presumably; from a performance perspective m:n threading, of which Erlang actors are a form, is generally agreed upon to lose some efficiency).

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#58

I agree with this article a lot. The layers over layers of software bloat increased with the hardware speed. The general trade-off is: We make the software as slow as possible to have the shortest development time with the least educated programmers. "Hey web dev intern, click me a new UI in the next two hours! Hurry!" This intern-clicks-a-new-UI works because we have a dozen layers of incomprehension-supporting-tech…

> More parallel CPUs need higher skilled devs working slower to utilize the chip's full potential.

Parallelism isn't so bad, you can use a functional style a lot of the time! E.g. Futhark, the accelerate framework, APLs....

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#59

Earlier quoted context omitted.

> no programs needed changing at all to get the full advantage of all the cores (barring some special cases) Some programs just don't have any available parallelism in them - no matter how many processes you split them up into. You need to build parallel algorithms and data structures. That's still often a research-level topic. Erlang's not going to magic parallelism out of thin air in a program that doesn't have any…

Kind of strawmanning here, no? Obviously sequential tasks are run sequentially. The OP isn't saying "magic" happens. Just, Erlang was written from the beginning to encourage parallelized code. Its whole programming model is around that. Unlike pretty much every language it was contemporary with, which assumed that the default was a single unit of execution (and that code running in parallel was the exception). Think…

> Kind of strawmanning here, no?

I don't think so? I'm responding to:

> no programs needed changing at all to get the full advantage of all the cores (barring some special cases)

Those 'special cases' are all the algorithms which aren't inherently parallel, which is most of them! How many algorithms inherently have a large amount of natural parallelism? I'd say it's very few.

For example no matter how you write MD5 on Erlang... it's not going to run in parallel is it?

Re: The free lunch is over: a fundamental turn toward concurrency in software (2005)

#60

Earlier quoted context omitted.

Kind of strawmanning here, no? Obviously sequential tasks are run sequentially. The OP isn't saying "magic" happens. Just, Erlang was written from the beginning to encourage parallelized code. Its whole programming model is around that. Unlike pretty much every language it was contemporary with, which assumed that the default was a single unit of execution (and that code running in parallel was the exception). Think…

> Kind of strawmanning here, no? I don't think so? I'm responding to: > no programs needed changing at all to get the full advantage of all the cores (barring some special cases) Those 'special cases' are all the algorithms which aren't inherently parallel, which is most of them! How many algorithms inherently have a large amount of natural parallelism? I'd say it's very few. For example no matter how you write MD5 o…

As I said before - 'Obviously sequential tasks are run sequentially.'

Innately sequential algorithms will still be sequential. Erlang already gave developers the tools, and incentive, to write things in a parallelized fashion. If a developer had a need that was directly sequential, it will, of course, still be sequential. If they had a need that would benefit from parallelism, they had the tools, the context, and the encouragement from the language itself to write it in parallel.

Again, I think you're strawmanning here. You kind of implied it yourself with your earlier statement, "Erlang's not going to magic parallelism out of thin air in a program that doesn't have any". Saying that implies you think that's the OP's claim.

As I said before, the OP wasn't claiming magic happens, and algorithms written serially suddenly become parallel. You can, of course, still write sequential code in Erlang, same as every language, because execution takes place in time. Just that the natural, ergonomic way of writing the language, of solving problems in the language, has always been parallel.

And if you mean "well, I run MD5 checksums on individual blocks of the file, and parallelize it that way" - yes, that is going to be one of those special cases the OP mentioned, if you didn't already optimize it that way due to memory pressure concerns. But if it's "run an md5 checksum against every file in this directory and check it against a manifest", the ergonomic approach in Erlang pre-multicore is to scatter/gather it across actors (since any one file might fail in strange ways and you want to isolate that failure and handle it predictably with a supervisor), and that would parallelize "for free", with no concurrency concerns. Unlike every other contemporary language.

Post reply on HN