Scala at scale at Databricks • Hacker News Board

Re: Scala at scale at Databricks

#131

Post by SOLAR_FIELDS » Fri, Dec 10, 2021, 5:10 PM UTC

tlarkworthy wrote: post #31

philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

As opposed to Gradle or SBT?

If I'm being honest here, a quite significant part of day-to-day backend JVM software development positions is dealing with the build tool. While I wouldn't outright reject a candidate for not being familiar with the latest standard toolchain (Gradle), I would certainly consider it as one of many factors - Gradle is pretty complicated, yes, but also quite powerful in what it is capable of achieving. It's not as eyebrow-raising as, say, if you sent in a coding challenge compiled using Ant (which would, in fact, be quite a fun discussion to have with a candidate) but it would certainly be something considered.

Re: Scala at scale at Databricks

#132

Post by philipkglass » Fri, Dec 10, 2021, 5:33 PM UTC

jongorer wrote: post #28

philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

The code base had multiple problems, none of which I would blame on category theory or Cats.

- Engineers had written higher abstractions seemingly just because they could. When I audited how internal libraries were used across our services, calling applications weren't making use of the advanced abstractions. I'm talking about things like using Cats to abstract across AWS S3 error handling. Cool, except that it wasn't needed because we never actually encountered the exotic compositions of failures anticipated by the libraries.

- The abstractions written for our own business logic were worse than abstracting over S3. They were premature. Our business logic had to change frequently because the end user experience was still evolving rapidly. Changes that violated previous assumptions and their corresponding abstractions took longer than they should have and/or led to very awkward code.

- At least at the time, tooling had more problems with the "advanced" code. The IntelliJ IDEA Scala plugin could not yet show how implicits were used. It couldn't find senders sending to an Actor the way it can easily find plain callers of a function. You would need to manually force a "clean" in certain modules before code changes would compile as expected. IDEs would also fail to flag code that couldn't compile, and incorrectly flag code that would compile, at a higher rate compared to plainer Scala.

I'm still glad that I have access to Cats, Akka, and other advanced parts of the Scala ecosystem. They're still used in a few places where their value is greater than their cost. Even in the plain code, I'm still very glad I have pattern matching, immutability-by-default, rich collections, map, flatMap, filter, fold, scan, find, etc. I have no plans to transition our company off Scala internally. If I were starting a greenfield project with myself as the sole developer, I'd probably be using Scala for that too. But I prefer to write a bunch of simple repetitive code first, then develop abstractions after it's clear what the commonalities are.

Re: Scala at scale at Databricks

#133

Post by ByteJockey » Fri, Dec 10, 2021, 5:52 PM UTC

AzzieElbab wrote: post #119

ByteJockey wrote: post #103

Earlier quoted context omitted.

Y'know what, I'll bite. I can do 10-15 pages. You have a link?

this one is not bad http://www.mas.ncl.ac.uk/~ndjw1/docs/djw-ctfp.pdf

Thanks. I appreciate the link. I'll take a look after work.

Re: Scala at scale at Databricks

#134

Post by 62951413 » Fri, Dec 10, 2021, 6:20 PM UTC

hyperbovine wrote: post #52

jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are wa…

The author of "The Science of Functional Programming" book (https://github.com/winitzki/sofp) makes an argument that CT is at best tangential to practicing FP.

Re: Scala at scale at Databricks

#135

Post by CraigJPerry » Fri, Dec 10, 2021, 6:28 PM UTC

sixminuteabs wrote: post #130

CraigJPerry wrote: post #108

Earlier quoted context omitted.

>> I can give them more freedom (because I know their programs won't compile if they don't line their types up In what meaningful way can you give them more freedom? When i think about how i spend my time with junior folks, it tends to be on things like: 1. Do they understand enough of our problem domain that i can begin to describe a problem that needs to be solved? 2. Have they understood the problem i described? H…

When I say more freedom, I primarily mean freedom to focus on the problem domain and freedom to contribute in a major way more quickly. Upon reflection, I realize that my expression about optimal solution was about optimality in modeling and understanding the problem domain vs. wrestling organizational programming pattern you're trying to learn / hold in your mind while also trying to solve the domain (which you don'…

>> they can get something "real" into prod much more quickly with almost no risk to the org

What are the most common risks to an org (originating from its code) and how do types get you down to almost no risk?

When i think of common risks, most of them don't even originate inside the process's address space so the type checker is completely disconnected from the risks (e.g. hey - you misunderstood what we wanted here...). For the risks that do originate directly from the code, it's stuff like hey - your solution is quadratic time, sure your code compiles and your tests all pass but i'm telling you this won't work under prod data volumes.

I'm trying to steelman your argument and the best case i can come up with is an IO monad.

Re: Scala at scale at Databricks

#136

Post by caoilte » Fri, Dec 10, 2021, 7:07 PM UTC

Zababa wrote: post #95

jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

There are others options besides Go and category theory. I'd say that for now, languages that are like ML are enough. They're coming in the mainstream with pattern matching, immutable records, etc. It'll take some time for people to learn this, learn to use it properly, teach it to others. And maybe some years later a monad will be a pattern as common and well-known as an iterator. But today isn't that day yet, and u…

The future is unevenly distributed.

There are plenty of companies where everyone has a very mature understanding of monads and where new joiners are coached to reach that level of understanding.

And there are plenty of places that aren't using source control, CI and unit tests.

Re: Scala at scale at Databricks

#137

Post by caoilte » Fri, Dec 10, 2021, 7:17 PM UTC

dikei wrote: post #63

amir734jj wrote: post #23

Earlier quoted context omitted.

In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :) Don't be negative about technology. Be positive.

> In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. It's funny that if you tell Cat-purist that Spark is functional-style, they would laugh at your face :)

That's an unhelpful thing to say. Most people agree that the Spark scala DSL is functional style. It's not as advanced in the style as other scala ecosystems but it has been a very good on ramp for people discovering functional style and is widely celebrated for that

Re: Scala at scale at Databricks

#138

Post by halfmatthalfcat » Fri, Dec 10, 2021, 8:02 PM UTC

mohanmcgeek wrote: post #129

halfmatthalfcat wrote: post #117

Earlier quoted context omitted.

Go is at the same level of complexity. JS I can agree with but we're starting to compare completely different things at that point. Type systems make things more complex at the benefit of safety.

Having used both Scala and Go for the past 6 years, I can tell that for the purposes of creating a database backed web API, they're definitely not at the same level of complexity. This isn't about the type system. This is about complicating things by making everything FP while getting nothing in return.

That's the great thing (and the point of many of the comments in this thread) about Scala - you don't have to do any FP with Scala if you so choose. You can operate in a completely OO fashion and gain all of the benefits of the language (primarily the type system) for free.

Re: Scala at scale at Databricks

#139

Post by random314 » Fri, Dec 10, 2021, 9:44 PM UTC

chrisjc wrote: post #86

random314 wrote: post #68

Earlier quoted context omitted.

As a counterpoint, I used scala at my previous job and it was the best coding experience and environment I ever experienced. Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark. The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after…

Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around? Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the da…

No. There was no transpiling to SQL. This was several years ago, before Spark SQL.

The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs.

HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop.

The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA.

In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.

Re: Scala at scale at Databricks

#140

Post by aweiland » Fri, Dec 10, 2021, 9:48 PM UTC

lmm wrote: post #50

dehrmann wrote: post #11

Earlier quoted context omitted.

Regarding 1, wouldn't Kotlin be a better choice?

I don't think so. Kotlin is an unmaintainable soup of features: it can match all the selling-point examples of Scala but only by adding ad-hoc special cases for each one. (For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approa…

This is a poor straw man argument. Spring boot is an option, but so is Ktor, or Dropwizard. You are arguing you preference as fact.