Scala at scale at Databricks
databricks.com
Scala at scale at Databricks
1–10 of 174 posts
Re: Scala at scale at Databricks
#2Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price.
How to deprecate this at scale?
The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api.
Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit.
Re: Scala at scale at Databricks
#3I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them.
I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case even when that made it hard to understand how they were used for the 2 actual cases we needed to handle. Akka, Cats, and Shapeless were all over the place.
Those earlier employees have churned off to other places and I have successively simplified the code they wrote that is still useful, while encouraging the use of no more language power than necessary in new development. Hands-On Scala Programming is the book I give new hires as a language introduction that shows the sort of style to be preferred. It's much more like super-powered Python than like Haskell.
I have written C, Fortran, JavaScript, Python, and Scala for money. When I started on Scala I had never written Java nor used any JVM language. I have come to really appreciate the rich ecosystem of JVM libraries, the instrumentation and profiling tools I get, and many aspects of the Scala language and standard library. I love Scala's collections and miss their power and ease when I'm writing Python. (Which I still do for certain scripting tasks and for accessing Python-ecosystem libraries.)
Re: Scala at scale at Databricks
#4Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…
Spark is really its own world that’s fairly separate from the Scala community at large. What makes it a ”terrible technology”?
Re: Scala at scale at Databricks
#5Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…
Spark is really its own world that’s fairly separate from the Scala community at large. What makes it a ”terrible technology”?
Scala moved the needle five years ago and made java better among other languages. Using scala helped me grow as an engineer. in my view, it has no place in an enterprise. I require a mature environment that favors consistency. It boils down to reducing complexity. I can barely handle the complexity growth due to the demands of the business, whether it be software engineering or people and these reasons do not justify scalas pluses. Ymmv.
Re: Scala at scale at Databricks
#6Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…
newer java has many similar features that scala had, so it seems moot to use it now.
unless you like ridiculously long compilation times and having libs tied to what version of the compiler you're using (serious WTF)
Re: Scala at scale at Databricks
#7Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…
newer java has many similar features that scala had, so it seems moot to use it now. unless you like ridiculously long compilation times and having libs tied to what version of the compiler you're using (serious WTF)
> it seems moot to use it now
It's true that some features of Scala have made it to Java, but I am not sure the gap between the two languages has shrunk much. For example while Java has added first-class functions and closures, it is still clumsy to attempt any serious functional programming approach with Java; Scala's type system is still way ahead; Scala compile to the JVM, JavaScript, and now native via LLVM.
> having libs tied to what version of the compiler you're using
It's never been that. However libraries used to be tied to the major version of Scala (for example 2.11, 2.12, 2.13) as major versions were not binary-compatible with each other.
Scala 3 addresses this thanks to the TASTY format which creates a strong foundation for binary compatibility. [1] You can use Scala 3 libraries with Scala 2.13 and, more importantly, Scala 2.13 libraries with Scala 3.
In short, Scala is not standing still.
[1] https://docs.scala-lang.org/scala3/guides/tasty-overview.htm...
Re: Scala at scale at Databricks
#8I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…
> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case even when that made it hard to understand how they were used for the 2 actual cases we needed to handle. Akka, Cats, and Shapeless were all over the place.
This hurts me a lot
Re: Scala at scale at Databricks
#9I completely agree with their pragmatic approach to Scala. Scala is a very powerful language, and you can do super crazy stuff. My approach to writing Scala code is similar:
1. Write the code like a better-Java/typesafe-Python (that is, imperative style).
2. Glance through the code. For collections, usually there's a more concise approach.
3. Read the code again. If you think what you did in #2 is making the code _too_ concise, you might need to think twice (or add comments).
Re: Scala at scale at Databricks
#10Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…
> Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit.
Yes it is. Deprecate using it and migrate away from it. I hate that these shitty overengineered Cucumber wannabees are what "Scala" means to most people.
It's a great language IMO, but you need to make sure fancy frameworks pay their weight. IMO Shapeless and Scalaz-or-equivalent do - replacing reflection or AOP with plain old compiled code, even complex code, is a huge win. But SBT and the mess of Scala testing libraries are just huge complexity for marginal benefit.