Scala at scale at Databricks • Hacker News Board

Re: Scala at scale at Databricks

#21

Post by EdwardDiego » Fri, Dec 10, 2021, 6:08 AM UTC

mohanmcgeek wrote: post #14

coffeecloud wrote: post #12

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

Why not just use Jackson? Sure, it doesn't have a clever DSL. But then, that's a feature.

Re: Scala at scale at Databricks

#22

Post by bamboozled » Fri, Dec 10, 2021, 6:09 AM UTC

mohanmcgeek wrote: post #14

coffeecloud wrote: post #12

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

I worked at a company full of Scala zealots and elites. It was insanely unproductive and people spent a lot of time trying to design around all the floors in Akka, trying to get Akka working in modern cloud computing environments and the like.

I think at some stage specialists from some Akka pro shop had to be air dropped in to try fix it all, but it didn't help much.

It always made me laugh how well all the services written in Python and Go worked and how easy they were to extend, operate and deploy comparatively.

Re: Scala at scale at Databricks

#23

Post by amir734jj » Fri, Dec 10, 2021, 6:14 AM UTC

waynesonfire wrote: post #2

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…

In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem.

I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :)

Don't be negative about technology. Be positive.

Re: Scala at scale at Databricks

#24

Post by lmm » Fri, Dec 10, 2021, 6:15 AM UTC

EdwardDiego wrote: post #21

mohanmcgeek wrote: post #14

Earlier quoted context omitted.

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

Why not just use Jackson? Sure, it doesn't have a clever DSL. But then, that's a feature.

Jackson has its own parallel "type registry" to make up for not being able to use something like Shapeless, because the problem still has to be solved. Getting a failure at compile time if there are any cases that wouldn't serialize properly is a huge advantage; a nontrivial proportion of production outages I've seen were Jackson.

Re: Scala at scale at Databricks

#25

Post by lmm » Fri, Dec 10, 2021, 6:17 AM UTC

mohanmcgeek wrote: post #14

coffeecloud wrote: post #12

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

The standard library is where modules go to die, just like in Python. And unlike Python, there's a good build/dependency manager available for Scala (Maven). Keep that stuff out of the core language, let it be libraries released on their own schedules.

Re: Scala at scale at Databricks

#26

Post by ackfoobar » Fri, Dec 10, 2021, 6:34 AM UTC

philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case

Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

Re: Scala at scale at Databricks

#27

Post by ackfoobar » Fri, Dec 10, 2021, 6:55 AM UTC

bamboozled wrote: post #22

mohanmcgeek wrote: post #14

Earlier quoted context omitted.

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

I worked at a company full of Scala zealots and elites. It was insanely unproductive and people spent a lot of time trying to design around all the floors in Akka, trying to get Akka working in modern cloud computing environments and the like. I think at some stage specialists from some Akka pro shop had to be air dropped in to try fix it all, but it didn't help much. It always made me laugh how well all the services…

My bad experience with Scala has more to do with the FP side (cats and all that). Akka has been quite pleasant to me, so I'm quite surprised by this.

Is the business problem not suited to the actor model in the first place?

Re: Scala at scale at Databricks

#28

Post by jongorer » Fri, Dec 10, 2021, 7:02 AM UTC

philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage.

Alternatively you can work in Go where braindead simplicity is the mandated norm.

Re: Scala at scale at Databricks

#29

Post by ackfoobar » Fri, Dec 10, 2021, 7:26 AM UTC

jongorer wrote: post #28

philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

I sympathize with your thinking. There are two "readabilities".

1. How easy it is for an average programmer to understand the code.

2. How understandable the code actually is if the reader is fluent with that prior knowledge.

If one finds the latter is much better than code written in conventional techniques, but people are scared off by the former, it can get pretty frustrating, and "people are stupid" may be a conclusion drawn.

---

But my experience is that those techniques do not yield better code.

And "Educate yourself" made me barf.

Re: Scala at scale at Databricks

#30

Post by grumpyprole » Fri, Dec 10, 2021, 7:30 AM UTC

foolfoolz wrote: post #19

msilb wrote: post #16

> If I had to describe the style of Scala written at Databricks, I’d put it at 50% Java-ish, 30% Python-ish, 20% functional This is interesting as in taking a different approach to most other Scala shops I’ve experienced so far where die-hard monoid/functors fanatics take over and make the code base completely incomprehensible to outsiders / new hires. This seems to be a much more sensible approach.

die hard functional scala is the exception not the rule. this isn’t really constrained to scala, language zealots can exist in any language. the real problem is a lack of understanding what maintainable code is

Yes, over-engineering is common in many languages, especially Java, where Spring and OOP patterns would be deployed everywhere. It is not a unique problem to Scala.