The free lunch is over: a fundamental turn toward concurrency in software (2005)

If you use Clojure than yes, software transactional memory is readily available. https://clojure.org/reference/refs

> that isn't really concurrent then

It is, using standard industry terminology.

Concurrency does not require that concurrent tasks are able to make independent progress at the same time, just that they are in some point of progress at the same time.

Back in 2012 when I was working on my PhD I wrote a paper about an example algorithm with very little inherent parallelism, where even if you have multiple concurrent tasks all making forward progress you may find it extremely hard to actually produce results in parallel, if you want to do a deep dive into this problem.

https://chrisseaton.com/multiprog2012/seaton-multiprog2012.p...

I wrote a poster about it as well for an easier introduction.

https://chrisseaton.com/research-symposium-2012/seaton-irreg...

The whole software vs hardware and developer vs runtime efficiency discussion is way more nuanced. No, the layers as they are currently are at least partially hindering productivity, because they all have bugs, rather painful limitations and are constantly changing under your feet. There is so much complexity, you don't have a chance to understand it (and the implications of it) completely from the click in a client's web browser all the way to the instructions executed on the server. This is not convenient and nobody (not even Google, Facebook and others) can handle the complexity, which manifests itself in bugs, weird behaviour, surprising security holes, other rare defects and more every day.

You don't want to manage memory manually unless you absolutely have to. Almost certainly, it is going to be way more demanding, you will make mistakes and the performance will not really be much better if you do whole system benchmarks for like 95% of applications. It is like using mining equipment and explosives to hang a picture on your concrete wall, it is just totally over the top. I am also not denying, there are use cases for something, where most of the responsibility is on you e.g. embedded or infrastructure software. Most people just aren't aware that they need the skill and understanding to reliably and most importantly responsibly wield it. In both cases, there is considerable room for better, more robust tools, and better methods/ practices and a great reduction in developer hubris.

Some have used better hardware for saving on development effort to some degree. There are applications that really need the extra performance to help users do their work in a more convenient way - not just developers. Some of the performance is lost due to systemic inefficiencies such as the web browser and most related technologies. E.g SVG rendering in web browsers is a mine field of bugs and bad performance, so you do the rendering yourself using canvas and JavaScript, which visually is the same thing but there is a lot more work for the developer and e.g. printing/ export into PDF (where you cannot just recalculate) is where you still kind of want the vector graphics capability in some form. Virtualization has enabled slicing of the cores to serve more customers with low cost and readily available compute. We also have way more users and different security expectations now than we used to in the past, so a single machine has to handle and check more stuff or we can use a single machine for stuff that would need a data centre or wouldn't be possible/ feasible before.

Most high-level languages implement their performance critical functions at a lower level. However, languages that rely on a GIL (Python and Ruby) or isolated memory spaces (V8) have an additional hurdle that if you want a model of concurrency with many threads acting simultaneously on a single shared memory space you have to do additional work.

For Python you either have to have your library pause execution of Python bytecode and do multithreaded things on your memory space, allocate an isolated memory arena for your multithreaded work, copy data in and out of it, and then spawn non-Python threads (see PEP 554 for an IRL example of this idea with the Python interpreter itself), or copy data to another process which can do the multithreaded things.

With PEP 554 (right now using the C API) I personally have no issue with multithreaded Python since the overhead of copying memory between interpreters is fast enough for my work but it is overhead.

> It is, using standard industry terminology.

Just to add some support here, concurrency definitely has very little to do with actual independence of progress. It has far more to do with encapsulation of knowledge (for an active agent, not a passive entity like an object [^]), and how different agents coordinate to exchange that knowledge.

An echo server + client is a concurrent system, despite having only one meaningful scheduling of actions over time (i.e. no interleaving or parallelism). Serialization and parallelism are both global properties, as they require a perspective "from above" to observe, while concurrency is a property local to each task.

I can appreciate everyone is saying upthread, but I've definitely found it valuable to think about concurrency in these terms.

[^] Even this is overselling it. You can easily have two logical tasks and one physical agent responsible for executing both of them, and this is still a concurrent system. Dataflow programming is a lot like this -- you're not really concerned with the global parallelism of the whole system, you're just describing a series of tasks over local knowledge that propagate results downstream.