Live data from Hacker News

Scala at scale at Databricks

databricks.com

51–60 of 174 posts

Re: Scala at scale at Databricks

#51

Earlier quoted context omitted.

I worked at a company full of Scala zealots and elites. It was insanely unproductive and people spent a lot of time trying to design around all the floors in Akka, trying to get Akka working in modern cloud computing environments and the like. I think at some stage specialists from some Akka pro shop had to be air dropped in to try fix it all, but it didn't help much. It always made me laugh how well all the services…

My bad experience with Scala has more to do with the FP side (cats and all that). Akka has been quite pleasant to me, so I'm quite surprised by this. Is the business problem not suited to the actor model in the first place?

Same here, Akka is really amazing

Re: Scala at scale at Databricks

#52

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are way more useful day to day. Once you’ve mastered calculus, linear algebra, probability theory, statistics, numerical analysis, optimization, algorithms, complexity theory, combinatorics, asymptotic analysis—sure, go pick up category theory.

Re: Scala at scale at Databricks

#54

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for.

I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening. And when you tried to tell them that abstractions don't come for free, but with a cost because they make code harder to understand, the answer for the last 25 years has always be the same, although rarely said straight: that it was only hard for -you- to understand, and not for the enlightened master himself.

I have seen more projects fail because of too much unneeded abstraction than by all other causes together. I even have seen companies go almost bankrupt because of projects engineered by lone wolfs where nobody understood the abstractions anymore except for the designer himself, and at some point he himself not really anymore either.

And the problem is that this mindset continues to be cultivated by CS books and conferences. Few people want to read a book that tells them that the secret of being productive as a team depends on mostly the culture in the team and the simplicity of the code, and not on the latest hyped framework, language or paradigm.

Re: Scala at scale at Databricks

#55

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for. I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were thos…

> I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening.

Agreed, and what did we get? dogmatic decrying of how OOP is completely useless and objectively bad, not too dissimilar from some of the comments on FP and Scala on here.

Re: Scala at scale at Databricks

#56

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

I worked at Databricks in the past and Scala is just a way to get things done there. Honestly, for me writing code day-to-day wasn't that different from Java once you understand how to structure your classes. Handling nullables with typed Options is pretty nice though, and I miss it in other languages now. Most people who join Databricks do not know Scala beforehand, it's just a useful tool.

Sure, sometimes progress is slower than you'd like, but that's the case in every large monorepo.

Re: Scala at scale at Databricks

#57
post #50

Earlier quoted context omitted.

Regarding 1, wouldn't Kotlin be a better choice?

I don't think so. Kotlin is an unmaintainable soup of features: it can match all the selling-point examples of Scala but only by adding ad-hoc special cases for each one. (For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approa…

> Kotlin is an unmaintainable soup of features

Are you sure you're not confusing Kotlin with Scala?

> For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approaches

And that is a problem how? Stick to one style.

> In practice Kotlin codebases still use magical incomprehensible reflection (Spring Boot)

https://github.com/spring-projects-experimental/spring-fu/tr...

> and magical compile-time manipulation (Kapt)

There's nothing magical about it.

Re: Scala at scale at Databricks

#58
I've seen a variety of Scala codebases from awesome to terrible (and a lot in between). Guess what, they all use the same Scala. Scala offers you many design choices that work well in skilled hands, but because it is so non-prescriptive, is is also very sensitive to individual talent and engineering culture. If you can't or don't want to maintain a team of strong developers, save yourself and use some other technology. But if you can, and you're working in a complex business domain, Scala can be hugely productive and result in very reliable software.

Just my counterpoint to all the negativity that people invariably feel compelled to pour out in any thread about Scala. Tons of devs use Scala productively without deep dives into monadic code golf, you know.

Re: Scala at scale at Databricks

#59
post #4

Earlier quoted context omitted.

Spark is really its own world that’s fairly separate from the Scala community at large. What makes it a ”terrible technology”?

Scala moved the needle five years ago and made java better among other languages. Using scala helped me grow as an engineer. in my view, it has no place in an enterprise. I require a mature environment that favors consistency. It boils down to reducing complexity. I can barely handle the complexity growth due to the demands of the business, whether it be software engineering or people and these reasons do not justify…

I think scala can absolutely account for that complexity, like it is not a one-way street. It can scale both up and down on expressivity. But I agree that often times it is not used that way.

Re: Scala at scale at Databricks

#60

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…

It saddens me to see that the HN crowd had been diluted to such an extent in terms of engineering quality that straight up luddism has become the norm, often accompanied with silly ad hominems.
Post reply on HN