What's Next for Scala

I very recently (about a year ago) joined a shop that is primarily into functional programming (FP) and I got my first exposure to Scala as well as F# through work projects. As a long time user of Java/OOP here are my thoughts on Scala and why it has so many things things going against it. Consider the following:

1) First, Scala is designed and developed by "ivory" tower academics primarily for research/PhD thesis purposes. Nothing wrong with that. But that also means language features are added not to solve a practical problem but mostly to show that "it can be done"/"prove a point". When you are awake at 3 AM early in the morning because a release made in the evening went awry and you are trying to grep though logs/code to see what caused the issue I can assure you that debates about "pure" versus "impure", Monads, Kleisli arrow will be further from your mind. So the problem that Scala seems to suffer from is that of "image".

2) Second, Scala is neither a fully functional (FP) nor object oriented (OOP). Support for mixed/multi-paradigm sounds nice on paper but let me ask this question: do you know of any hard core FP programmer who is excited about inheritance or abstract classes or other OOP features? Vice-versa the same thing can be said about many OOP programmers lukewarm response towards many functional features of Scala. I believe that support for mixed/multiple paradigms has hindered large scale adoption of Scala. The language seems to be caught at cross-roads not knowing which way to advance next.

3) Third is lack of niche. True, I absolutely love "Spark" and primarily use it to extract/manipulate data from HDFS/Hive storage but I am gradually gravitating towards "Python" libraries due to their ease of use. What about better Java? Kotlin check. What about sidecar systems used by DevOps? Go check. What about a better designed system language? Rust check. The days of pure general purpose programming languages are truly over and Scala will be in great peril if it fails to expand into newer niches. Without broadening its programmer base it will finally run out of people that can design newer language features in light of ever advancing progresses in processor, OS and memory architectures.

Just my two cents.

I think a major reason Scala flourished was a big reason it has struggled as well: it opened the doors for many people to new and fundamentally different programming paradigms such as actor systems and functional programming. Many people got excited and implemented these new paradigms in production systems without regard to the immaturity of their own grasp of the concepts (think of how many years of imperative/OO programming you had under your belt before feeling confident of setting the style for a project), the learning curve they were non-consensually committing their coworkers to, and the hiring, productivity, and morale burden they were saddling their employers with. From the management point of view, allowing Scala in the door meant running the risk of waking up one day with a bunch of Scalaz in the codebase, one grinning functional programmer happy as a pig in shit, and the rest of the team shouting and cursing.

Likewise, on top of the real problems created by ill-conceived adoption, the irrational backlash against new paradigms is real as well. I'm old enough to remember the fuss over OO programming. Many experienced, successful coders were scared that they were about to become personally obsolete. Some OO proponents insinuated — and panicked veteran coders believed — that a large number of procedural programmers would have to retire or switch careers because they were too old or not intellectually nimble enough to adapt to object-oriented programming. Major companies such as Sun responded to these fears with a marketing blitz to prove their their OO technology was simple and easy to learn and would herald a new age of programmer productivity. Programmer culture was more driven by magazines and books in those days and therefore much more easily influenced by industry marketing. The "simplicity" of Java was an invention of Sun marketing, a successful self-fulfilling prophecy as Java became so familiar that similarity to it became millions of programmers' benchmark of simplicity.

There has been no total war marketing blitz for functional or actor-based programming from a company with the same stature that Sun had in the 1990s. The emotional reaction, the resentment at feeling a step behind the future, the anxiety about whether you will ever be as comfortable and confident in the new paradigm, operate virtually in a vacuum, or worse, in the echo chamber of the internet. Whatever latent fear of functional programming a person feels, the prospect of using Scala will bring it to the fore. The only countervailing force is the (usually lone) individual pushing adoption, and they are unlikely to be up to the challenge. There is a tension between programmers who are more oriented towards business problems, or who just want to get their job done and go home, and the programmers who get excited about different and potentially better ways to express programs. They both think of themselves as humble and results-oriented and have a hard time accepting each other as being equally humble and results-oriented in their own way. They both experience the other as snobbish, condescending, and threatening to their self-image as good engineers.

Sadly, the root of the problem is that Scala is expressive and powerful enough to be a natural vehicle for these new paradigms. An engineering organization should not fear expressive power, because they should have a culture that is incompatible with a single engineer (or a small cabal) committing their peers to a learning curve without consultation, debate over the cost and payoff, and buy-in from the team. Unfortunately, there seems to be an assumption that beyond choosing a language, you have no ability to consciously choose a style. I.e., Scala-as-a-better-Java (the least imaginative approach, but not a bad one) is an impossibility. This is grounded in the assumption that any aspect of a language that can be abused, will be abused, and that engineers cannot make collective decisions about how they will work together, which are odd assumptions to make about the behavior of highly paid professionals. As someone who wrote C++ professionally for many years, I can testify that it is possible for engineers to conform to coding standards that restrict them to a strict subset of the expressive power of a language. However, it sometimes requires management authority to bring people in line -- and perhaps this is a deal-breaker in an industry where teams of engineers are supposed to be entirely self-managing.

Clojure did have quite a bit of hype 5-10 years ago and a bit of slowdown in 2017-2018 where this meme of "Clojure is dead" probably started.

The Scala team pulled out all the stops with Dotty. It's a real tour de force, on a par with the sea change of C++11. Even with the plaudits he's earned, Martin Odersky is for me in the top three language designers of our time. (I'm sure I'm being unfair to other members of his team with this comment, though.)

Dotty's union types are a thing of beauty, its implicits are really second to none, and _so_ damn useful. Their decision to embrace whitespace sensitivity, while simplifying the syntax for predominant use-cases (again, union types!) betokens flexibility, and a desire to swing for the rafters.

The only thing that saddens me somewhat is that Scala Native does feel like a bit of a second class citizen. The `stackalloc` feels a little sad (coming from C++ -- it should be inverted!). I wonder if it's possible for Typesafe to leverage Scala Native in a manner akin to Python with Numba/Cython. With whitespace indentation - native code inlined and FFI'd automatically - right there with the rest of the nice JVM stuff.

I can't resist citing my email from Feb '07 asking (politely) why we needed a 'new' keyword. Dotty apparently removes the need for that keyword. Should have listened earlier, team Scala ;)

http://www.scala-archive.org/quot-new-quot-keyword-td1989540...

Agreed. I switched from scala to python and oh man what a relief.

Scala is a heavy weight languange. By heavyweight I mean,

1. Even intellij struggles to figure out the meaning of implicits in the given context. And then there are macros.

2. Way too many theoretical concepts, monads, monoids , isomorphisms ect that make you feel stupid. There is always a nagging feeling that you don't 'get it' when you program in scala. Here is an example of scala conference in 2020:

https://scaladays.org/2020/berlin/schedule

Features talks like these.

> a. Composition with Functional Effects

> b. Scala 3 and Abstract Algebra

> c. Describable, typesafe computations for the less monadically inclined

> d. Understanding Scala's Type System

> e. Roll your own Effect System

3. Every code base is written in its own flavor of scala.

I was spending absurd amount of time "learning scala", now i look back at that and cringe.