From First Principles: Why Scala? • Hacker News Board

Re: From First Principles: Why Scala?

#141

Post by MrPowers » Thu, Feb 11, 2021, 6:15 PM UTC

dominotw wrote: post #38

aynyc wrote: post #15

Is Scala used outside of Spark and Akka? Spark is my company chosen analytic engine, Scala was/is the main language for majority of the production workload. In the last two years or so, more devs are writing PySpark and SparkSQL jobs than Scala.

def mirrors my experience too. Vast majority of spark jobs were easily ported to sql/dbt and the remaining ones are in pyspark. I used to use a lot of scala spark in backend data processing in 2016 but now its almost down to zero. scala is real big impediment to making data processing accessible to general public in your company. order of preference now at my company is, 1. sql 2. pyspark 3. java spark 4. scala spark…

I've rolled out Scala based Spark interfaces to non-programmers in Databricks notebooks, so it's definitely possible, but only if you stick with the basic language features.

Here's a more detailed PySpark vs Scala comparison in case folks are interested: https://mungingdata.com/apache-spark/python-pyspark-scala-wh...

I think Scala Spark (using 10% of the language features) is a better technical decision (because it provides huge benefits like fat JARs, shading, better text editor support, etc), but the worse overall choice for most organizations because people are generally terrified of Scala.

They'd rather do nothing than write Scala code. I can empathize with their position.

Re: From First Principles: Why Scala?

#142

Post by joshlemer » Thu, Feb 11, 2021, 6:16 PM UTC

yw3410 wrote: post #105

Kototama wrote: post #95

Earlier quoted context omitted.

It makes the code impure (that is with side-effects). One can also argue that if you can encode the potential errors in the type they are not exceptional. It's not that bad though, because you get much less of these kind of errors than in a typical Java program (for example).

Yes, it is a side-effect - and does make the code impure; but it's an /exceptional/ flow like OOM, infinite looping, stack overflows which also break equational reasoning. My argument is that as a programmer you /choose/ the base lemmas which you're comfortable with - with an exception you're saying for a large swathe of your code, it will assume a lemma. When is /does/ break you get to point the finger at it with a…

This is spot on! Way too often in Scala codebases, which tend to wrap IO calls in Future or similar is that you'll have chains of side effecting futures like

    readFromDb()
      .flatMap(x => doSomethingElse(x))
      .flatMap(x => doSomethingElseAgain(x))
      .flatMap(x => onceAgain(x))

And this all gets passed up the call stack to one generic thing that basically does nothing in the case of error, maybe logging it and that's it. It's tempting to write code like this because it's so easy and looks so clean and readable, but in reality it is not how you build reliable software because what one must do in response to a failure from the first call is not the same as what one must do for a failure of the second call, or third or fourth, but this flatMap().flatMap().flatMap() style seduces the programmer into thinking they've handled errors when really they've just lumped the entire workflow as one big chain that can fail at any point and in practice usually just ignore all error cases.

One strategy I find does slightly mitigate this is to use future.transformWith[S](f: Try[T] => Future[S]). This at least presents to you the opportunity to think about error handling a bit more consciously, but I still find that the chaining operator approach nudges you away from thinking about error handling because it becomes quickly difficult to read.

It's probably to do with how limited the control structures are when dealing in Futures. You don't have while/if-else/etc, you must lift all your control into flatMaps and iterations are only doable via recursive calls with explicit accumulator state (yuck!) and nesting. So whereas a properly thoughtful treatment of error cases in synchronous code may take 20 or 30 lines of legible code, this gets tranformed in the async style to 20 or 30 levels of nested callbacks and recursive calls which becomes unreadable.

Re: From First Principles: Why Scala?

#143

Post by wtetzner » Thu, Feb 11, 2021, 6:17 PM UTC

vemv wrote: post #122

lihaoyi wrote: post #106

Earlier quoted context omitted.

> - don't pretend the JVM and JS have identical runtimes when it comes to concurrency or numerics. Clojure is a language, not a platform abstraction. > - Emit efficient, optimally minifiable javascript code, at the cost of offering something less traditionally lisp-y when it comes to eval, macros, and other forms of code loading. Those are still possible, but some discipline is imposed. This is exactly the kind of th…

The JVM offers real threads, and JS not. The JVM offers nanosecond-precision time measurement, and JS not. JS has a single type for representing numbers. And so on. How can those possibly be abstracted away and unified? If doing so, what would have the solution have to do with type inference at all? i.e., this is simply a difference in how one approaches platform interop (raw vs abstracted/unified). Clojure occasiona…

> The JVM offers real threads, and JS not.

How does this make a difference to the Clojure interface to threads?

After all, if you're running your JVM on hardware that doesn't support parallelism, you're not getting parallelism anyway.

> JS has a single type for representing numbers.

The comment you're replying to already covered that:

> For example, the reason Scala.js can provide exact semantics for numerics while maintaining high performance is precisely because it knows what exactly types things are, and can use that information to optimize the generated JS in a semantics-preserving way

> Clojure occasionally offers cross-platform abstractions (e.g. core.async) but it's not its main philosophy.

I can understand providing different libraries for different platforms, but language semantics should remain (as much as is possible), e.g. numerics.

Re: From First Principles: Why Scala?

#144

Post by dboreham » Thu, Feb 11, 2021, 6:17 PM UTC

agbell wrote: post #4

Great Analysis! I agree that languages are becoming more like Scala. I'm not sure about the JIT part though. Aren't many of the recent language success stories now about AOT compiled languages like Go, Rust and Swift?

Scala is AOT. The JIT part is just to speed up execution of the machine code its compiler targets, it not being the native ISA typically.

Re: From First Principles: Why Scala?

#145

Post by brabel » Thu, Feb 11, 2021, 6:19 PM UTC

MrPowers wrote: post #50

dominotw wrote: post #40

Earlier quoted context omitted.

> Hard to win technical arguments with Scala geniuses that like using complicated language features. I was on a team building good old crud apps using monads, monoids, categories, combinators, effects cats, seamless and bunch of other nonsense that i've now purged from my brain. You could've easily mistaken our team for a programming language research group at a university. This is literally the number one reason i w…

In the Spark world, you can use a tiny subset of the Scala features and enjoy huge productivity gains over the other language APIs (Java & Python). Those productivity gains are wiped out as more crazy language features get used. I don't think the super complex language features should be removed. Li's libs do some crazy stuff under the hood, but provide a clean, Python-like public interface. Most devs aren't that goo…

> Lots of folks would love Scala codebases that only use 10% of the available language features and none of the complex frameworks.

They already do it: it's called Kotlin.

Re: From First Principles: Why Scala?

#146

Post by wtetzner » Thu, Feb 11, 2021, 6:20 PM UTC

the_af wrote: post #26

mwcampbell wrote: post #20

I dabbled with Scala several years ago, but I've been using Kotlin for a JVM-based project and am happy with it. My main reason for choosing Kotlin is smoother interop with the JVM world. For example: - Scala adds an Option type, whereas Kotlin adds nullability checking for existing object types. - Scala has its own convention for getters and setters, whereas Kotlin automatically turns JVM getters and setters into pr…

> Scala defines its own set of collection classes [...] One (arguably) negative consequence of this is that Scala's collections don't play nice with Hibernate and similar ORMs. Due to the way they work, both Scala and Hibernate wanted to "take over" your collections, and obviously they cannot do so at the same time. Of course, it's arguable whether this is a flaw with Scala or with Hibernate, but for the programmers…

> PS: if I remember correctly, early Scala adopters had reached the consensus that ORMs didn't play nice with Scala's FP style anyway, and so using Hibernate was a mistake. Possibly this attitude changed later.

Having used Hibernate in a Java project, I would argue using Hibernate is always a mistake. Way too much complexity/brokenness for basically no benefit.

Re: From First Principles: Why Scala?

#147

Post by wtetzner » Thu, Feb 11, 2021, 6:21 PM UTC

thelittlenag wrote: post #63

the_af wrote: post #26

Earlier quoted context omitted.

> Scala defines its own set of collection classes [...] One (arguably) negative consequence of this is that Scala's collections don't play nice with Hibernate and similar ORMs. Due to the way they work, both Scala and Hibernate wanted to "take over" your collections, and obviously they cannot do so at the same time. Of course, it's arguable whether this is a flaw with Scala or with Hibernate, but for the programmers…

The two major SQL libraries in the Scala ecosystem these days are Doobie ( https://tpolecat.github.io/doobie/ ) and Slick ( https://scala-slick.org/ ). With Doobie you manually write your queries, and then map the results into the objects in your domain model. Nothing is generated for you. OTOH, nothing is hidden and you are free to write queries as optimized and specialized as you need. The real selling point of Doo…

Isn't JOOQ also usable with Scala?

Re: From First Principles: Why Scala?

#148

Post by the_af » Thu, Feb 11, 2021, 6:23 PM UTC

lihaoyi wrote: post #116

the_af wrote: post #25

Earlier quoted context omitted.

You're not obligated, but much like Maven is the build tool for Java, sbt is the build tool for Scala. You use alternatives at your own peril. This means you won't be able to understand other people's builds, for example. I know some Scala projects use Maven instead. Whether that's a good idea is debatable!

> You use alternatives at your own peril. This means you won't be able to understand other people's builds, for example. The tradeoff is with SBT, you won't be able to understand your own builds! (This is kind of a joke, but not really...)

Sadly, you're completely right. It's not a joke: it's happened to me. I still have nightmares about sbt.

Re: From First Principles: Why Scala?

#149

Post by joshlemer » Thu, Feb 11, 2021, 6:23 PM UTC

randmeerkat wrote: post #111

Kototama wrote: post #100

Earlier quoted context omitted.

Tell that to Amazon, Apple, Netflix, Cognitect and all enterprises that produce reliable and performant software with it everyday.

I don’t know of anyone at those companies using clojure and I haven’t seen any tech blogposts by those companies about using clojure. If clojure exists at those companies I imagine it’s a small very niche team.

One very public example of a Clojure shop through and through is NuBank out of Brazil, who employs some 700 "Clojure developers" according to them. In fact they are so committed to Clojure they bought Cognitect last year.

Re: From First Principles: Why Scala?

#150

Post by randmeerkat » Thu, Feb 11, 2021, 6:26 PM UTC

jinfiesto wrote: post #121

yw3410 wrote: post #81

Earlier quoted context omitted.

I am continually perplexed at the swallowing of exceptions in certain FP communities. Result types, Either et cetera make it extremely easy to swallow errors by flatMapping thoughtlessly losing the context of where they were - you typically /want/ the call stack when you hit into an exceptional flow.

You shouldn't swallow errors like that in FP either. You can trap the error in an effect type and throw it at the top level of your app when your effect type gets run after all of the code that processes that error value has a chance to recover. See something like Zio for an example, though you can do similar things without Zio.

If the team uses Zio, or maybe they’ve gotten lost in the religious war of Scalaz, vs Cats, vs Zio... One of the major issues I have with Scala is how fragmented the community is. Everyone has an opinion on how something should be done and everyone thinks that everyone else is wrong.