Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

81–90 of 174 posts

Re: Scala at scale at Databricks

#81

Post by LandR »

jongorer wrote: post #28
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

This is development nowadays, hire people fresh out if am eigh week javascript boot camp and everyone has to write code they can understand.

Instead of bringing the juniors up, we drag the more experienced guys down to their level.

The result is really shit basic code everywhere. or you end up using a language like go where there's basically only for, if and arrays and writing code in it is miserable and tedious.

Re: Scala at scale at Databricks

#82

Post by [deleted] »

tlarkworthy wrote: post #31
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

[deleted]

Re: Scala at scale at Databricks

#84

Post by AzzieElbab »

snidane wrote: post #13
coffeecloud wrote: post #12

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

Same. It's a fairly accurate predictor of a company downturn after getting sunk in total complexity and ensuing certain employee churn. The only reason for Scala adoption was Spark 10 years ago. Scala/Spark engineers were pulling the top buck. With pyspark and other cloud big data options on the scene, no one sane is using scala for this anymore.

Spark wasnt even released 10 years ago :) I think Twitter, Linkedin, and some others gave Scala adaptation big push. Twitter in particular, when they killed their fail whale by moving to Scala/JVM

Re: Scala at scale at Databricks

#85

Post by AzzieElbab »

noelwelsh wrote: post #79
MrPowers wrote: post #64

Earlier quoted context omitted.

> Scala offers you many design choices that work well in skilled hands I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala c…

> Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite. It's hard to make generalizations like this that are meaningful. There are a number of different Scala ecosystems, as you probably know. Spark is really a world of its own. Lightbend's libraries form their own ecosystem, as do Typelevel's and ZIO's. My own opinion is that using algebraic da…

I always find it funny how ppl will whine about Scala's complexity but then when Scala's "complex" features get re-implemented in java(usually in some bastardized fashion) they are god-givens

Re: Scala at scale at Databricks

#86

Post by chrisjc »

random314 wrote: post #68
waynesonfire wrote: post #2

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes…

As a counterpoint, I used scala at my previous job and it was the best coding experience and environment I ever experienced. Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark. The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after…

Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around?

Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.

Re: Scala at scale at Databricks

#87

Post by sireat »

joelbluminator wrote: post #77
MrPowers wrote: post #64

Earlier quoted context omitted.

> Scala offers you many design choices that work well in skilled hands I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala c…

In what way is Scala a better Python? The 2 seem nothing alike to me.

Scala is almost as terse as Python for data munging/wrangling tasks (unlike say C#,C++ or Java)

Scala is almost as much joy to write data munging tasks as Python (unlike say C#, C++, Java, and I have to say Golang).

Chaining multiple maps and filters is so much more pleasurable than writing 4 nested loops with multiple ifs inside.

The better part is that you can reliably deploy Scala unlike Python.

We just had yet another discussion on the mess that Python deployment is: https://news.ycombinator.com/item?id=29439149

Re: Scala at scale at Databricks

#88

Post by hocuspocus »

_vertigo wrote: post #44
jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

If all you do is add numbers together, is it anti-intellectual to question why one needs to grok Principia Mathematica first rather than just doing some arithmetic? In other words, do you solve problems that actually require category theory or are you just navel gazing?

If you've ever done a map reduce on a dataset you've used category theory, whether knowingly or otherwise.

You don't need more than highschool maths to understand the category theory behind common typeclasses.

Re: Scala at scale at Databricks

#89

Post by hocuspocus »

Fiahil wrote: post #78
nikitaga wrote: post #58

I've seen a variety of Scala codebases from awesome to terrible (and a lot in between). Guess what, they all use the same Scala. Scala offers you many design choices that work well in skilled hands, but because it is so non-prescriptive, is is also very sensitive to individual talent and engineering culture. If you can't or don't want to maintain a team of strong developers, save yourself and use some other technolog…

> But if you can, [...] Scala can be hugely productive and result in very reliable software. Yes, but, in my experience, the productivity gains are not that big. Strong developpers can and _will_ fare well in any technical environments, therefore choosing Scala over anything else won't give your team any advantage over a "simpler" technology. It's actually riskier to pick Scala, because you will increase your recruit…

> Save yourself, don't use Scala, pick Rust.

That's an hilariously bad advice if you think hiring competent developers and onboarding juniors is hard in Scala.

Most things companies do on the JVM, especially at scale, are going to be a lot harder in Rust and its ecosystem.

Re: Scala at scale at Databricks

#90

Post by sireat »

MrPowers wrote: post #35

Scala is a great language when Li Haoyi ecosystem of libraries is used. His libs do crazy Scala stuff under the hood, but expose a clean end user interface. The problem with Scala is that most ppl don't use his libs. Here's a blog post I wrote on reading / writing JSON using one of Li's libs: https://mungingdata.com/scala/read-write-json/ There's now a completely sensible option for JSON in Scala. But there are also…

I seriously love Li Haoyi Scala libraries especially the JSON one.

The hilarious thing is/was that Scala tutorials/books would show writing JSON parser as one examples. Yeah Scala makes it easy to write a JSON parser but you still do not want to roll your own.

Case in point Odersky's Programming in Scala.

Post reply on HN