Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

161–170 of 174 posts

Re: Scala at scale at Databricks

#161

Post by esarbe »

asimpletune wrote: post #69

I love Scala by the way and have written hugs code based on it. However the reason why Scala can encourage bad code from smart people is that it allows people to max out their budget for writing smart code. It’s like, let’s say it requires 2x more intelligence budget to read/debug than it is to write code. So if we all have more or less an intelligence budget of like 10, maybe some it’s 12 or others even 15-20, and y…

> For most of my career in Scala, when I had conflict with other engineers, it was over this kind of stuff. They insisted on using cats or scalaZ or writing these crazy complicated, overly generic abstractions, and it’s like…

While I kinda agree that you have to be careful with your abstractions, I see no harm getting familiar with some of the concepts that these libraries use.

Being able to just implement a Monoid and then use it to do nested aggregation in maps is pretty neat, for example.

Re: Scala at scale at Databricks

#162

Post by EdwardDiego »

lmm wrote: post #153
EdwardDiego wrote: post #150

Earlier quoted context omitted.

I'll be honest, I had a quick look at Shapeless, and couldn't see how you'd apply to ensuring an entity serialized correctly. But I'm keen to learn more :) I always prefer compile time to run time errors.

What Shapeless gives you is the ability to treat value-like types - case classes and sealed traits - generically. And so you can do "walk the object graph" style stuff at compile time - particularly, typeclass derivation. https://circe.github.io/circe/codecs/auto-derivation.html is one popular library that does this - note that the `.asJson` will fail at compile time if you add a member to Greeting or Person whose ty…

Okay, that's pretty cool. Cheers :)

Re: Scala at scale at Databricks

#163

Post by esarbe »

hyperbovine wrote: post #52
jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are wa…

I found category theory to be helpful in giving a vocabulary for some very simple and essential things that otherwise get lost. My goto example are Monoids and Semi-Groups, which are almost mindbogglingly simple but pop up everywhere.

Category theory gives you the precise language to talk about such stuff and helps to communicate and talk about more complex concepts and that's always a plus.

Re: Scala at scale at Databricks

#164

Post by dudul »

Prunkton wrote: post #76
KptMarchewa wrote: post #74

>Zero usage of “archetypical” Scala frameworks: Play, Akka, Scalaz, Cats, ZIO, etc. This is the key to their success.

could you elaborate please? I'm especially interested in the Play/Akka part. So what would the alternatives look like, using common Java Frameworks instead or none at all?

Depends on what you're trying to do. For example I think that Lightbend is no longer planning to maintain Play. I would probably go with Spring as a replacement.

For Akka, do you really need it? You have to be building a pretty specific kind of system to need it over simpler constructs built into the language.

Re: Scala at scale at Databricks

#165

Post by StreamBright »

valenterry wrote: post #147
StreamBright wrote: post #80

Earlier quoted context omitted.

Same. Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem". I think F# is doing much better in this category. Scala can surprise you many unexpected ways. https://www.youtube.com/watch?v=v1wrWQcqLpo

F# has some nice features that scala is lacking, such as a more terse syntax for ADTs or type providers. But other than that, Scala is miles ahead. And if you want to do pure FP, then Scala is the only option. Also about that video, almost all of the points mentioned are either not available in other languages like F# anyways, or they already fixed in the latest Scala version.

> And if you want to do pure FP, then Scala is the only option.

I don't think that Scala is especially good for "pure FP".

I also do not want to do "pure FP".

Re: Scala at scale at Databricks

#166

Post by StreamBright »

esarbe wrote: post #159
StreamBright wrote: post #80

Earlier quoted context omitted.

Same. Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem". I think F# is doing much better in this category. Scala can surprise you many unexpected ways. https://www.youtube.com/watch?v=v1wrWQcqLpo

> Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem". Could you elaborate? Because that's not been my experience at all. I have found Scala to provide many tools to wrap around the idiosyncrasies of the underlying imperative that they are barely noticeable anymore.

I found F# really good at with this -> "wrap around the idiosyncrasies of the underlying imperative".

In the video Paul Phillips elaborates on many subject much better than I could ever do.

I also found the .net ecosystem libraries better, even though there are fewer libraries available.

Some links that I found great on the subject:

https://www.reddit.com/r/scala/comments/4246qc/heres_why_sca...

https://techblog.bozho.net/i-dont-like-scala/

To be honest with you my social circle of software engineers gave up Scala long time ago and we do not even consider it to try it ever again. Most of the projects we do we simple use Java for JVM (since version 8 there are many useful features that you can use and have relatively terse code that is also easy to read) for the .net ecosystem we use C# of F# depending on how relaxed the engineering team and the stakeholders about it.

For data engineering (ETLs, glue code, etc.) Python works. Since Spark has a Python interface it is also not needed to use Scala there.

For frontend obviously Scala was probably never a serious option.

There is no use for Scala for me or for the companies I work for. The last time I tried Scala it was horrendous. SBT? Seriously? Binary incompatibility between minor versions? I could go on an on of the collection of anti-patterns I have encountered. Since I have limited time and there are a ton of other software environments that we can use and have much better experience why on Earth would I want to waste time on Scala ever again?

Re: Scala at scale at Databricks

#167

Post by AzzieElbab »

scalaIsKindaMeh wrote: post #142
AzzieElbab wrote: post #123

Earlier quoted context omitted.

Scala compilation + graalvm compilation = very long breaks :)

Interesting, in my experience long compile times in Scala equates with a bad code base. And invariably involves an uber object with a whole lot of implicit conversations and is imported on every other source file.

I don’t know about bad code but a lot of derivations that send compiler into proof loops usually is what is causing slowdowns. You can write the same code by hand, pick your poison. Honestly, scalac is probably still faster than typescript compiler and somehow no one complaining about ts

Re: Scala at scale at Databricks

#168

Post by valenterry »

StreamBright wrote: post #165
valenterry wrote: post #147

Earlier quoted context omitted.

F# has some nice features that scala is lacking, such as a more terse syntax for ADTs or type providers. But other than that, Scala is miles ahead. And if you want to do pure FP, then Scala is the only option. Also about that video, almost all of the points mentioned are either not available in other languages like F# anyways, or they already fixed in the latest Scala version.

> And if you want to do pure FP, then Scala is the only option. I don't think that Scala is especially good for "pure FP". I also do not want to do "pure FP".

> I don't think that Scala is especially good for "pure FP".

I meant when compared to F#. Of course there are other languages for FP as well.

> I also do not want to do "pure FP".

That's your choice!

Re: Scala at scale at Databricks

#169

Post by justsaying9 »

esarbe wrote: post #163
hyperbovine wrote: post #52

Earlier quoted context omitted.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are wa…

I found category theory to be helpful in giving a vocabulary for some very simple and essential things that otherwise get lost. My goto example are Monoids and Semi-Groups, which are almost mindbogglingly simple but pop up everywhere. Category theory gives you the precise language to talk about such stuff and helps to communicate and talk about more complex concepts and that's always a plus.

[flagged]

Re: Scala at scale at Databricks

#170

Post by justsaying9 »

esarbe wrote: post #157

Scala is an amazing language I love writing software in. The tools it offers to model business domains, the power of expressive types make it next-to-none when it comes to shape, remodel and refactor code. Then there's the ecosystem. Sure, you have the whole of the Java ecosystem, but then there are also tons and tons of Scala libraries that are so well designed (Quick shout-out to Lihaoyi here; your libraries are am…

Thanks for the advice. Given that you exposed your warped, racist ideological beliefs in other threads, this helps me to understand that Scala probably isn't something I want to involve myself with under any circumstances. Your judgment about good or bad at best cannot be trusted, and at worst is a contrary indication warning me to stay the hell away.
Post reply on HN