Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

111–120 of 174 posts

Re: Scala at scale at Databricks

#111

Post by hocuspocus »

perceptronas wrote: post #98

In my personal experience, Scala in 2021 is de facto best language for business logic. Effects are easy, parallel code is beautiful and easy to reason about. Ecosystem is big and no other languages comes close to productivity I have with Scala. For example, its very fast and easy to prototype code with ADTs and scale that code to big codebase. I rarely have to run my code and if I do - it most of the time works as in…

> With tagless final, Kleisli and so on.

Tagless final is a good way to enforce the principle of least power, which has many advantages when onboarding inexperienced developers. It can become unwieldy if abused, but that's true for any style.

And if it really bothers you, then you can use your concrete "super monad" everywhere. It's the only way with ZIO, but you can also do that with other effects.

Regarding Kleisli, it's such a broad abstraction that you'll find examples everywhere, whether explicit or not. In practice, even with heavy usage of libraries like fs2 or http4s, it's mostly an internal implementation detail, in most cases you'll only see type aliases.

Re: Scala at scale at Databricks

#112

Post by mohanmcgeek »

halfmatthalfcat wrote: post #104
mohanmcgeek wrote: post #14

Earlier quoted context omitted.

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

Those are the least confusing libraries in the Scala ecosystem lol. Slick is just like any ORM-light framework. Akka-HTTP abstracts away all the Akka-specific things and you just create routes that return Futures essentially. How is spray-json complex? You're just using macros against your case classes to create (de)serializers. If anything, you should be saying things like Shapeless provide the extra complexity.

> Those are the least confusing libraries in the Scala ecosystem

Now that says something about the Scala ecosystem. Doesn't it?

I'm comparing this to how easily I can do the same task in nodejs and Go.

Re: Scala at scale at Databricks

#113

Post by noelwelsh »

MrPowers wrote: post #96
noelwelsh wrote: post #79

Earlier quoted context omitted.

> Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite. It's hard to make generalizations like this that are meaningful. There are a number of different Scala ecosystems, as you probably know. Spark is really a world of its own. Lightbend's libraries form their own ecosystem, as do Typelevel's and ZIO's. My own opinion is that using algebraic da…

I work on Dask now and the creator summarizes his preferred code style nicely in this post: https://matthewrocklin.com/blog/work/2018/01/27/write-dumb-c... . I like working on simple codebases like this. The only Typelevel library I've used is frameless, which I blogged about here: https://mungingdata.com/apache-spark/frameless-typed-dataset... . Don't think the lib is really usable. Typelevel stuff is not what I'm r…

Agreed on Frameless. I think Frameless was an experiment that didn't pan out. I don't think Scala 2 is capable of implementing a usable system that does what Frameless attempted to do (a typed data frame). Scala 3 might.

I find what most people mean when they say "simple" is really "familiar". When I first started teaching most programmers weren't familiar with first class functions and they were "complex". Now almost everyone has been exposed to them. Similarly I'm seeing more students are learning FP at university and thus find FP techniques "simple".

Re: Scala at scale at Databricks

#114

Post by eweise »

Fiahil wrote: post #78
nikitaga wrote: post #58

I've seen a variety of Scala codebases from awesome to terrible (and a lot in between). Guess what, they all use the same Scala. Scala offers you many design choices that work well in skilled hands, but because it is so non-prescriptive, is is also very sensitive to individual talent and engineering culture. If you can't or don't want to maintain a team of strong developers, save yourself and use some other technolog…

> But if you can, [...] Scala can be hugely productive and result in very reliable software. Yes, but, in my experience, the productivity gains are not that big. Strong developpers can and _will_ fare well in any technical environments, therefore choosing Scala over anything else won't give your team any advantage over a "simpler" technology. It's actually riskier to pick Scala, because you will increase your recruit…

I haven't used Rust but my understanding is that memory management is explicit, which doesn't sound great for general business applications. I've used Java, Python, JS, Kotlin, Go and Scala, and found Scala to be the best at representing domain models in a type safe and readable way.

Re: Scala at scale at Databricks

#115

Post by mohanmcgeek »

the_only_law wrote: post #55
misja111 wrote: post #54

Earlier quoted context omitted.

Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for. I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were thos…

> I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked…

> dogmatic decrying of how OOP is completely useless and objectively bad

And just who were the ones pushing this? The people peddling FP.

The people in this thread are not pushing anything. They're just sharing their experiences with Scala and how unproductive it is to deal with FP zealots.

Re: Scala at scale at Databricks

#116

Post by eweise »

waynesonfire wrote: post #2

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…

Well, you know, you can just use Junit with Scala. That's what I do.

Re: Scala at scale at Databricks

#117

Post by halfmatthalfcat »

mohanmcgeek wrote: post #112
halfmatthalfcat wrote: post #104

Earlier quoted context omitted.

Those are the least confusing libraries in the Scala ecosystem lol. Slick is just like any ORM-light framework. Akka-HTTP abstracts away all the Akka-specific things and you just create routes that return Futures essentially. How is spray-json complex? You're just using macros against your case classes to create (de)serializers. If anything, you should be saying things like Shapeless provide the extra complexity.

> Those are the least confusing libraries in the Scala ecosystem Now that says something about the Scala ecosystem. Doesn't it? I'm comparing this to how easily I can do the same task in nodejs and Go.

Go is at the same level of complexity. JS I can agree with but we're starting to compare completely different things at that point. Type systems make things more complex at the benefit of safety.

Re: Scala at scale at Databricks

#118

Post by vasco »

abernard1 wrote: post #46
ackfoobar wrote: post #26

Earlier quoted context omitted.

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

> Scala the language is quite fine, but it attracts this kind of programmers. It does. I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language. I've interviewed >100 Scala engineers and if the panel thought they were the kind of person that just wanted to understand FP instead of solve problems, they were rejected. I am now retired, but…

I read this and I get so tempted but then I load up the homepage and see "JVM" and regain my senses. You almost got me.

Re: Scala at scale at Databricks

#119

Post by AzzieElbab »

ByteJockey wrote: post #103
AzzieElbab wrote: post #92

Earlier quoted context omitted.

I hear you. However the amount of category theory you need to understand 99.99% of FP is really really minimal. 10-15 pages of well written text

Y'know what, I'll bite. I can do 10-15 pages. You have a link?

this one is not bad http://www.mas.ncl.ac.uk/~ndjw1/docs/djw-ctfp.pdf

Re: Scala at scale at Databricks

#120

Post by jvican »

vasco wrote: post #118
abernard1 wrote: post #46

Earlier quoted context omitted.

> Scala the language is quite fine, but it attracts this kind of programmers. It does. I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language. I've interviewed >100 Scala engineers and if the panel thought they were the kind of person that just wanted to understand FP instead of solve problems, they were rejected. I am now retired, but…

I read this and I get so tempted but then I load up the homepage and see "JVM" and regain my senses. You almost got me.

Well, Scala compiles to JavaScript and native through Scala.js and Scala Native. Plus you can also compile to native with GraalVM Native which works very well.
Post reply on HN