Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

41–50 of 174 posts

Re: Scala at scale at Databricks

#41

Post by MrPowers »

derN3rd wrote: post #33

It amazes me that such big projects use monorepos. My company also though about doing so, because we have a few typescript projects that share a common module with business logic and everything lags behind when the common module was upgraded, but I don't know if I should sell them to switch over to a mono repo as well. My biggest pro argument would be the speed up in development and review time, because everything is…

Navigating well designed, huge monorepo Scala codebases is surprisingly easy. Spark is a great example of a huge monolith that's relatively easy to work with considering the complexity of the app.

If you follow the design principles outlined in this doc (https://www.lihaoyi.com/post/StrategicScalaStylePrincipleofL...), Scala projects are really refactorable, which is nice.

But I've also worked on lots of large Scala codebases that are completely incomprehensible and painful to change. Scala allows for devs to make beautiful monoliths, but that's not usually what happens.

Re: Scala at scale at Databricks

#42

Post by the_only_law »

_vertigo wrote: post #39
tlarkworthy wrote: post #31

Earlier quoted context omitted.

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

How did that conversation go? “We like your solution but did you seriously just use Maven?” What did they say exactly?

Yeah that’s got to be up there on the list for dumbest technical related rejection I’ve ever heard of, particularly if the code was done well otherwise.

Re: Scala at scale at Databricks

#43

Post by derN3rd »

MrPowers wrote: post #41
derN3rd wrote: post #33

It amazes me that such big projects use monorepos. My company also though about doing so, because we have a few typescript projects that share a common module with business logic and everything lags behind when the common module was upgraded, but I don't know if I should sell them to switch over to a mono repo as well. My biggest pro argument would be the speed up in development and review time, because everything is…

Navigating well designed, huge monorepo Scala codebases is surprisingly easy. Spark is a great example of a huge monolith that's relatively easy to work with considering the complexity of the app. If you follow the design principles outlined in this doc ( https://www.lihaoyi.com/post/StrategicScalaStylePrincipleofL... ), Scala projects are really refactorable, which is nice. But I've also worked on lots of large Scal…

We use TypeScript not Scala, but I've already found some other TS projects that use monorepos for the exact same reason I would do.

Still searching for pros/cons with current toolings etc, as most articles about monorepos are 3-4y old and therefore outdated in some points

Re: Scala at scale at Databricks

#44

Post by _vertigo »

jongorer wrote: post #28
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

If all you do is add numbers together, is it anti-intellectual to question why one needs to grok Principia Mathematica first rather than just doing some arithmetic?

In other words, do you solve problems that actually require category theory or are you just navel gazing?

Re: Scala at scale at Databricks

#45

Post by linspace »

jongorer wrote: post #28
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

For most tasks programming it's more an engineering discipline than an intellectual pursuit. In that context complexity must always be justified.

I know, and have been told, to have written hard to read code. In my case it's usually vectorized code in numpy or a C or cython extension for really hot code. But I always have a good reason, usually performance when it matters.

It's usually said that premature optimization is the root of all evil, but it's nothing compared against premature abstraction. Optimization at least is local in its nature while abstraction tends to expand all over a code base, and when the assumptions made for the abstraction no longer hold people is still forced to keep dancing for a music that no longer plays.

"Educate yourself" assumes that your interests are everyone's interests but this is a vast field and not everyone has interest nor time to learn about category theory. Most probably your coworkers are quite intelligent. If they are not interested in your idea it may be that it's not appropriate or maybe not correctly framed, or not mature.

Re: Scala at scale at Databricks

#46

Post by abernard1 »

ackfoobar wrote: post #26
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

> Scala the language is quite fine, but it attracts this kind of programmers.

It does. I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language. I've interviewed >100 Scala engineers and if the panel thought they were the kind of person that just wanted to understand FP instead of solve problems, they were rejected.

I am now retired, but Scala is still my goto choice for engineering software. It checks all the boxes on concurrency, bulk I/O, ecosystem. In the hands of a sane and pragmatic practitioner, it's hugely productive.

Re: Scala at scale at Databricks

#47

Post by Toine »

wiradikusuma wrote: post #9

I completely agree with their pragmatic approach to Scala. Scala is a very powerful language, and you can do super crazy stuff. My approach to writing Scala code is similar: 1. Write the code like a better-Java/typesafe-Python (that is, imperative style). 2. Glance through the code. For collections, usually there's a more concise approach. 3. Read the code again. If you think what you did in #2 is making the code _to…

4. Switch to Kotlin or Java

Re: Scala at scale at Databricks

#48

Post by ngc248 »

jongorer wrote: post #28
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

This ain't anti-intellectualism. Scala fanatics (more like FP fanatics who use scala) write overly completed code just for purity's sake. Its ok, but it creates a maintenance nightmare and a huge hiring problem.

Re: Scala at scale at Databricks

#49

Post by dxxvi »

dehrmann wrote: post #11
wiradikusuma wrote: post #9

I completely agree with their pragmatic approach to Scala. Scala is a very powerful language, and you can do super crazy stuff. My approach to writing Scala code is similar: 1. Write the code like a better-Java/typesafe-Python (that is, imperative style). 2. Glance through the code. For collections, usually there's a more concise approach. 3. Read the code again. If you think what you did in #2 is making the code _to…

Regarding 1, wouldn't Kotlin be a better choice?

I think no, because, IMO, Kotlin doesn't have a good pattern matching like Scala.

Re: Scala at scale at Databricks

#50

Post by lmm »

dehrmann wrote: post #11
wiradikusuma wrote: post #9

I completely agree with their pragmatic approach to Scala. Scala is a very powerful language, and you can do super crazy stuff. My approach to writing Scala code is similar: 1. Write the code like a better-Java/typesafe-Python (that is, imperative style). 2. Glance through the code. For collections, usually there's a more concise approach. 3. Read the code again. If you think what you did in #2 is making the code _to…

Regarding 1, wouldn't Kotlin be a better choice?

I don't think so. Kotlin is an unmaintainable soup of features: it can match all the selling-point examples of Scala but only by adding ad-hoc special cases for each one. (For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approaches). In practice Kotlin codebases still use magical incomprehensible reflection (Spring Boot) and magical compile-time manipulation (Kapt), so you have the worst of both Java and Scala worlds.
Post reply on HN