Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

91–100 of 174 posts

Re: Scala at scale at Databricks

#91

Post by AzzieElbab »

wiseowise wrote: post #38
ackfoobar wrote: post #26

Earlier quoted context omitted.

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

> who both understands those advanced concepts and is not a FP fanatic. Oxymoron.

I speak monads. I am a 50 yrs old high school dropout. Cool thing is - just because you are not seeing FP structures in non FP focused languages doesn't mean they aren't there.

Re: Scala at scale at Databricks

#92

Post by AzzieElbab »

hyperbovine wrote: post #52
jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are wa…

I hear you. However the amount of category theory you need to understand 99.99% of FP is really really minimal. 10-15 pages of well written text

Re: Scala at scale at Databricks

#93

Post by flowcont »

linspace wrote: post #45
jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

For most tasks programming it's more an engineering discipline than an intellectual pursuit. In that context complexity must always be justified. I know, and have been told, to have written hard to read code. In my case it's usually vectorized code in numpy or a C or cython extension for really hot code. But I always have a good reason, usually performance when it matters. It's usually said that premature optimizatio…

I'm stealing that phrase: "Premature abstraction is the root of all evil". It's hard for me sometimes to justify in a code review why an abstraction is not required (yet) when someone has put some effort into it.

Re: Scala at scale at Databricks

#94

Post by sixminuteabs »

We develop the majority of our code in scala at my company (in fintech).

There seems to be an experience in this thread that training in Scala is difficult. For anyone who is considering and wants another perspective - I have found that junior engineers tend to get up to speed more quickly in Scala than in other languages and that I can give them more freedom (because I know their programs won't compile if they don't line their types up and our use of the type system very immediately forces them to deal with the possibility that an operation can fail, e.g.). I can simply look at types and give them a nudge in the right direction (heyoo -fp either joke-). Yes they suffer a little in the beginning, but are usually shipping code within a few weeks then their trajectory really begins to rocket. It's very cool to watch them grow to building complex systems in a complex domain so quickly. As they grow the ease of refactoring means that it is easy to change their prior work as they gain more understanding, so for our management the penalty of allowing them to learn in the wild about something that's suboptimal is pretty low - they will come back and fix it. Of course what I'm saying above is really about FP and type systems and an implicit (heyoo -scala joke-) presence of thought about how our systems work top to bottom and how we train people. I think that is also required no matter what tools you use or what you're doing in any organization?

Re: Scala at scale at Databricks

#95

Post by Zababa »

jongorer wrote: post #28
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

There are others options besides Go and category theory. I'd say that for now, languages that are like ML are enough. They're coming in the mainstream with pattern matching, immutable records, etc. It'll take some time for people to learn this, learn to use it properly, teach it to others. And maybe some years later a monad will be a pattern as common and well-known as an iterator. But today isn't that day yet, and until now you have to collaborate with your peers. You also have to assume that you may be wrong about some of this stuff, and keep an open mind for FP alternatives. For example, I think algebraic effects aren't part of category theory, yet they're an exciting new feature that might have an impact in "industrial" languages one day.

Re: Scala at scale at Databricks

#96

Post by MrPowers »

noelwelsh wrote: post #79
MrPowers wrote: post #64

Earlier quoted context omitted.

> Scala offers you many design choices that work well in skilled hands I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala c…

> Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite. It's hard to make generalizations like this that are meaningful. There are a number of different Scala ecosystems, as you probably know. Spark is really a world of its own. Lightbend's libraries form their own ecosystem, as do Typelevel's and ZIO's. My own opinion is that using algebraic da…

I work on Dask now and the creator summarizes his preferred code style nicely in this post: https://matthewrocklin.com/blog/work/2018/01/27/write-dumb-c.... I like working on simple codebases like this.

The only Typelevel library I've used is frameless, which I blogged about here: https://mungingdata.com/apache-spark/frameless-typed-dataset.... Don't think the lib is really usable. Typelevel stuff is not what I'm referring to when I say better Python. I'll try to make another blog to clarify.

I think Scala is a better language than Python. But for some reason it just seems like Scala is full of mind bending code that I can't understand. Full disclosure: I am way less smart than lots of the Scala ppl I've worked with.

Re: Scala at scale at Databricks

#97

Post by MrPowers »

sireat wrote: post #90
MrPowers wrote: post #35

Scala is a great language when Li Haoyi ecosystem of libraries is used. His libs do crazy Scala stuff under the hood, but expose a clean end user interface. The problem with Scala is that most ppl don't use his libs. Here's a blog post I wrote on reading / writing JSON using one of Li's libs: https://mungingdata.com/scala/read-write-json/ There's now a completely sensible option for JSON in Scala. But there are also…

I seriously love Li Haoyi Scala libraries especially the JSON one. The hilarious thing is/was that Scala tutorials/books would show writing JSON parser as one examples. Yeah Scala makes it easy to write a JSON parser but you still do not want to roll your own. Case in point Odersky's Programming in Scala.

Yea, there was a JSON parser in the Scala standard lib at one point and it was then deprecated & removed. Guess it's easy to make a "hello world" JSON lib, but maybe not the best addition to the standard lib, haha.

Re: Scala at scale at Databricks

#98

Post by perceptronas »

In my personal experience, Scala in 2021 is de facto best language for business logic. Effects are easy, parallel code is beautiful and easy to reason about. Ecosystem is big and no other languages comes close to productivity I have with Scala.

For example, its very fast and easy to prototype code with ADTs and scale that code to big codebase. I rarely have to run my code and if I do - it most of the time works as intended because of the type system. Its like magic. In other languages if you want to write code without bugs you are stuck with TDD or a lot of tests. With Scala usually few tests are enough, everything else encoded in its expressive type system.

I would say the only big negative I see is using Scala as Haskell. With tagless final, Kleisli and so on. If you do - it becomes very hard to onboard new people and I am not sure the positives outweigh the negatives.

Re: Scala at scale at Databricks

#99

Post by tharne »

mhzsh wrote: post #4
waynesonfire wrote: post #2

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…

Spark is really its own world that’s fairly separate from the Scala community at large. What makes it a ”terrible technology”?

I wouldn't say spark is terrible, but it is a sort of a "reverse bicycle". If you put a spark codebase down for a week or two and pick it back up, it feels like you're learning everything all over again.

Re: Scala at scale at Databricks

#100

Post by gregkerzhner »

wiseowise wrote: post #38
ackfoobar wrote: post #26

Earlier quoted context omitted.

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

> who both understands those advanced concepts and is not a FP fanatic. Oxymoron.

Oxymonad?
Post reply on HN