Live data from Hacker News

Scala at scale at Databricks

databricks.com

71–80 of 174 posts

Re: Scala at scale at Databricks

#71

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are wa…

But Category Theory is different to all the other examples you gave, as it describes program composition. i.e. building larger programs from smaller ones. Computer scientists have been searching for the "mathematics of program construction" for a long time, desperately trying to make the process more formal. Category Theory is what many seem to have arrived at. There are even books about it, e.g. "The Algebra of Programming".

Most computer science graduates are well schooled in algorithms, complexity, linear algebra etc. However, the vast majority of their time commercially will not be spent implementing some core algorithm, but instead gluing systems together and consuming/writing APIs. Any mathematics that helps them better accomplish this would be good to add to their toolbox.

Re: Scala at scale at Databricks

#72

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

I also prefer to use Maven to compile Scala code - it's much faster, better integrated with IDEs, etc.

Re: Scala at scale at Databricks

#73

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…

In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :) Don't be negative about technology. Be positive.

Scala + Spark is absolutely the sane face of Scala. Try using stuff like cats (basically Haskel ported to Scala) to experience the deep end of the pool.

Re: Scala at scale at Databricks

#76

>Zero usage of “archetypical” Scala frameworks: Play, Akka, Scalaz, Cats, ZIO, etc. This is the key to their success.

could you elaborate please? I'm especially interested in the Play/Akka part. So what would the alternatives look like, using common Java Frameworks instead or none at all?

Re: Scala at scale at Databricks

#77

I've seen a variety of Scala codebases from awesome to terrible (and a lot in between). Guess what, they all use the same Scala. Scala offers you many design choices that work well in skilled hands, but because it is so non-prescriptive, is is also very sensitive to individual talent and engineering culture. If you can't or don't want to maintain a team of strong developers, save yourself and use some other technolog…

> Scala offers you many design choices that work well in skilled hands I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala c…

In what way is Scala a better Python? The 2 seem nothing alike to me.

Re: Scala at scale at Databricks

#78

I've seen a variety of Scala codebases from awesome to terrible (and a lot in between). Guess what, they all use the same Scala. Scala offers you many design choices that work well in skilled hands, but because it is so non-prescriptive, is is also very sensitive to individual talent and engineering culture. If you can't or don't want to maintain a team of strong developers, save yourself and use some other technolog…

> But if you can, [...] Scala can be hugely productive and result in very reliable software.

Yes, but, in my experience, the productivity gains are not that big. Strong developpers can and _will_ fare well in any technical environments, therefore choosing Scala over anything else won't give your team any advantage over a "simpler" technology. It's actually riskier to pick Scala, because you will increase your recruitment difficulty by a factor of 100-1000 (less developers available in the pool, even less when you only consider skilled developers, and huge barriers to teach and bring up to speed anyone that haven't worked in Scala before). What do you think will happen when the "good" developers are gone ? A rewrite in Python.

Save yourself, don't use Scala, pick Rust.

Re: Scala at scale at Databricks

#79

I've seen a variety of Scala codebases from awesome to terrible (and a lot in between). Guess what, they all use the same Scala. Scala offers you many design choices that work well in skilled hands, but because it is so non-prescriptive, is is also very sensitive to individual talent and engineering culture. If you can't or don't want to maintain a team of strong developers, save yourself and use some other technolog…

> Scala offers you many design choices that work well in skilled hands I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala c…

> Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite.

It's hard to make generalizations like this that are meaningful. There are a number of different Scala ecosystems, as you probably know. Spark is really a world of its own. Lightbend's libraries form their own ecosystem, as do Typelevel's and ZIO's.

My own opinion is that using algebraic data types (sealed traits, final case classes) with a sprinkling of type classes (implicits) where needed is pretty easy to work with, and that's the style that Typelevel is using. It is mostly inline with Li's "Least Power" post, though using IO pervasively instead of Future.

Is Typelevel's approach a better Python? I don't know. I don't really know what that means, and I'm not convinced that Python is a great model to follow. I see you posting along similar lines in many Scala threads on HN, so you seem quite invested in the language. I think it would help make your point clearer if wrote up a bit more of your thoughts, perhaps as a blog post.

Re: Scala at scale at Databricks

#80

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

Same. Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem".

I think F# is doing much better in this category.

Scala can surprise you many unexpected ways. https://www.youtube.com/watch?v=v1wrWQcqLpo

Post reply on HN