Scala at scale at Databricks • Hacker News Board

Re: Scala at scale at Databricks

#121

Post by [deleted] » Fri, Dec 10, 2021, 4:03 PM UTC

ackfoobar wrote: post #26

philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

[deleted]

Re: Scala at scale at Databricks

#122

Post by jvican » Fri, Dec 10, 2021, 4:08 PM UTC

Fiahil wrote: post #106

hocuspocus wrote: post #89

Earlier quoted context omitted.

> Save yourself, don't use Scala, pick Rust. That's an hilariously bad advice if you think hiring competent developers and onboarding juniors is hard in Scala. Most things companies do on the JVM, especially at scale, are going to be a lot harder in Rust and its ecosystem.

> That's an hilariously bad advice Absolutely not! I'm serious, and I stand by it. Rust ecosystem is like a better Python. You have to consider why someone is willing to move away from Python and into a foreign language: Developer supply, performance, safety, ecosystem, political reasons. I can't cover all cases so I'm going to focus on the one that makes the most sense _from my personal point of view_. I do AI Softw…

You can use Scala with VS Code just in the same way you can do it with Rust. Tooling support for Scala is world-class in many ways. I think your knowledge is pretty outdated.

Re: Scala at scale at Databricks

#123

Post by AzzieElbab » Fri, Dec 10, 2021, 4:15 PM UTC

jvican wrote: post #120

vasco wrote: post #118

Earlier quoted context omitted.

I read this and I get so tempted but then I load up the homepage and see "JVM" and regain my senses. You almost got me.

Well, Scala compiles to JavaScript and native through Scala.js and Scala Native. Plus you can also compile to native with GraalVM Native which works very well.

Scala compilation + graalvm compilation = very long breaks :)

Re: Scala at scale at Databricks

#124

Post by random314 » Fri, Dec 10, 2021, 4:20 PM UTC

chrisjc wrote: post #86

random314 wrote: post #68

Earlier quoted context omitted.

As a counterpoint, I used scala at my previous job and it was the best coding experience and environment I ever experienced. Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark. The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after…

Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around? Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the da…

Most of this was Hadoop. Which executes mapreduce jobs written in sql on the jvm. The scalding jobs executed faster, except for one job for which we had to stick with Apache pig.

Spark jobs were native RDDs before the SQL Spark era.

Re: Scala at scale at Databricks

#125

Post by to11mtm » Fri, Dec 10, 2021, 4:29 PM UTC

lmm wrote: post #36

ackfoobar wrote: post #27

Earlier quoted context omitted.

My bad experience with Scala has more to do with the FP side (cats and all that). Akka has been quite pleasant to me, so I'm quite surprised by this. Is the business problem not suited to the actor model in the first place?

I've never seen a case where the actor model actually helped with the business problem. Every time I saw Akka used, replacing it with straightforward Futures-and-for/yield code was an improvement. Cats et al are frequently abused - 90% of the time rather than using a complex tool to solve a complex problem you want to think about it a bit and simplify it. But when you need them they do things that you can't do any ot…

> I've never seen a case where the actor model actually helped with the business problem.

I've inherited and built some actor based applications and libraries, both in Erlang and .NET (via Akka.NET). Usually these systems are:

- heavily Domain/event driven

- have 'funnel points' (example; bids on specific cars)

- have strong (but not extreme) requirements around consistent, fast response.

- be reasonably maintainable without a whole lot of gotchas

In all cases the actor model was the easiest way to solve the problems presented. That last point may be a bit contentious, but what every shop I've been at has found that while sometimes domain evolution results in extra boilerplate code to handle different versions of events, it is usually not the same sort of nightmare to do larger redesigns of the system as it ages and needs evolve.

> Every time I saw Akka used, replacing it with straightforward Futures-and-for/yield code was an improvement.

I'll admit I've over-used Akka.NET from time to time. I'd say the most overkill case is 'I need background workers running on timers with automatic recovery.'

Re: Scala at scale at Databricks

#126

Post by iskander » Fri, Dec 10, 2021, 4:38 PM UTC

My experience with "Scala at scale" was extremely unsatisfactory and I found Databricks to give very unhelpful responses to basic performance problems arising from the intersection of Scala and Spark, bordering on gaslighting. Memory bloat is one of the most pernicious issues, you have to handle large data very carefully to not end up with massive memory footprints that outstrip any available hardware (and causes hard to debug cascading failures).

Re: Scala at scale at Databricks

#127

Post by throwaway47292 » Fri, Dec 10, 2021, 4:51 PM UTC

haha all this scala hate as if huge javascript or c or whatever at scale are the best

the reality is, we are in this spaghetti together, so pick your noodle and move on

its just computers, dont take it so seriously :)

Re: Scala at scale at Databricks

#128

Post by zackangelo » Fri, Dec 10, 2021, 4:56 PM UTC

joelbluminator wrote: post #77

MrPowers wrote: post #64

Earlier quoted context omitted.

> Scala offers you many design choices that work well in skilled hands I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala c…

In what way is Scala a better Python? The 2 seem nothing alike to me.

People often compare them because they're the two best supported languages for interacting with Spark (and its ecosystem).

Re: Scala at scale at Databricks

#129

Post by mohanmcgeek » Fri, Dec 10, 2021, 5:02 PM UTC

halfmatthalfcat wrote: post #117

mohanmcgeek wrote: post #112

Earlier quoted context omitted.

> Those are the least confusing libraries in the Scala ecosystem Now that says something about the Scala ecosystem. Doesn't it? I'm comparing this to how easily I can do the same task in nodejs and Go.

Go is at the same level of complexity. JS I can agree with but we're starting to compare completely different things at that point. Type systems make things more complex at the benefit of safety.

Having used both Scala and Go for the past 6 years, I can tell that for the purposes of creating a database backed web API, they're definitely not at the same level of complexity.

This isn't about the type system. This is about complicating things by making everything FP while getting nothing in return.

Re: Scala at scale at Databricks

#130

Post by sixminuteabs » Fri, Dec 10, 2021, 5:07 PM UTC

CraigJPerry wrote: post #108

sixminuteabs wrote: post #94

We develop the majority of our code in scala at my company (in fintech). There seems to be an experience in this thread that training in Scala is difficult. For anyone who is considering and wants another perspective - I have found that junior engineers tend to get up to speed more quickly in Scala than in other languages and that I can give them more freedom (because I know their programs won't compile if they don't…

>> I can give them more freedom (because I know their programs won't compile if they don't line their types up In what meaningful way can you give them more freedom? When i think about how i spend my time with junior folks, it tends to be on things like: 1. Do they understand enough of our problem domain that i can begin to describe a problem that needs to be solved? 2. Have they understood the problem i described? H…

When I say more freedom, I primarily mean freedom to focus on the problem domain and freedom to contribute in a major way more quickly. Upon reflection, I realize that my expression about optimal solution was about optimality in modeling and understanding the problem domain vs. wrestling organizational programming pattern you're trying to learn / hold in your mind while also trying to solve the domain (which you don't have to do as much with scala's compiler)

And in terms of morale, I view almost the opposite of what you're saying. Because types and the compiler enforce correctness in the program and compatibility within our systems I can have a relatively junior person thinking more about the domain and knowing that they can get something "real" into prod much more quickly with almost no risk to the org. They can move on to bigger blocks of work more quickly. Solving real, big problems and making impact gets engineers jazzed up. Regarding compilation, I find the "this won't compile" phase is almost non-existent in practice. People quickly adapt from "this thing doesn't compile [insert sad longing for python]" to "this thing won't compile - somewhere there's a misalignment with what I said I'm going to do (via types) and what I'm actually doing, [insert silent acknowledgement that they would have shipped an error in python and not known it]."

I'm not seeing where the gap is between A and B except that in option A it appears that there is no compiler involved, which per the above, means that I have to ask them to focus more on exercising mental discipline about coding conventions (boring, low value add work) and also I can't allow them to contribute as quickly because the risk they threw an exception somewhere they shouldn't have (e.g.) is higher.