Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

141–150 of 174 posts

Re: Scala at scale at Databricks

#141

Post by scalaIsKindaMeh »

abernard1 wrote: post #46
ackfoobar wrote: post #26

Earlier quoted context omitted.

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

> Scala the language is quite fine, but it attracts this kind of programmers. It does. I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language. I've interviewed >100 Scala engineers and if the panel thought they were the kind of person that just wanted to understand FP instead of solve problems, they were rejected. I am now retired, but…

> I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language.

I think this is a bit contradictory as it is phrased. Learning the language is essential to use it effectively. Otherwise one is better off with a language already known. And it is very possible to build things to learn the language and still keep it sane.

If we went back to let's say 2014 "when the cake pattern was the solution of all problems in world" I think I'd agree with your statement, even back then when I was a bit of a Scala fanatic.

Re: Scala at scale at Databricks

#142

Post by scalaIsKindaMeh »

AzzieElbab wrote: post #123
jvican wrote: post #120

Earlier quoted context omitted.

Well, Scala compiles to JavaScript and native through Scala.js and Scala Native. Plus you can also compile to native with GraalVM Native which works very well.

Scala compilation + graalvm compilation = very long breaks :)

Interesting, in my experience long compile times in Scala equates with a bad code base. And invariably involves an uber object with a whole lot of implicit conversations and is imported on every other source file.

Re: Scala at scale at Databricks

#143

Post by scalaIsKindaMeh »

tlarkworthy wrote: post #31
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

That's silly but unfortunately this type of silliness is the norm in the industry.

Re: Scala at scale at Databricks

#144

Post by scalaIsKindaMeh »

misja111 wrote: post #54
jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for. I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were thos…

> before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening.

Ah! Just write a quine and be done with it!

> I have seen more projects fail because of too much unneeded abstraction than by all other causes together. I even have seen companies go almost bankrupt because of projects engineered by lone wolfs where nobody understood the abstractions anymore except for the designer himself, and at some point he himself not really anymore either.

"cake pattern" on one side and scalatest on the other.

As for

> I’m constantly surprised to see this type anti-intellectualism.

You hardly need any Category Theory to theory to understand type theory or mostly important Scala's type system. This the exact knowledge and allows you use it effectively. Yeah, the way he puts it makes those who like to understand the theory behind everything look bad.

Re: Scala at scale at Databricks

#145

Post by jim-jim-jim »

the_only_law wrote: post #34
ackfoobar wrote: post #29

Earlier quoted context omitted.

I sympathize with your thinking. There are two "readabilities". 1. How easy it is for an average programmer to understand the code. 2. How understandable the code actually is if the reader is fluent with that prior knowledge. If one finds the latter is much better than code written in conventional techniques, but people are scared off by the former, it can get pretty frustrating, and "people are stupid" may be a conc…

So, I’ll probably get shit on for this, but I really cannot wait to leave this industry. I hate how “best tool for the job” really just means “what we can have a large hiring pool of people who’ve only ever learned one programming style and possibly only one language of” or “for legacy reasons this is the only choice”. Yes I’m familiar with reality, I just hate it.

Yes, it's disappointing but hardly surprising to see the phrasing of management parroted by so many developers. It's their job to worry about hiring, deadlines, etc. It's yours to make sure things don't shit the bed at one in the morning, especially if you're going to be paged about it. There are some languages that make it very easy for grads to deliver features, and others that are far more likely to be correct at compile time. The "best tool" is born from these competing demands; those too meek to engage with this inherently adversarial process don't do themselves or anybody else any favors.

Re: Scala at scale at Databricks

#146

Post by zaphirplane »

AzzieElbab wrote: post #85
noelwelsh wrote: post #79

Earlier quoted context omitted.

> Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite. It's hard to make generalizations like this that are meaningful. There are a number of different Scala ecosystems, as you probably know. Spark is really a world of its own. Lightbend's libraries form their own ecosystem, as do Typelevel's and ZIO's. My own opinion is that using algebraic da…

I always find it funny how ppl will whine about Scala's complexity but then when Scala's "complex" features get re-implemented in java(usually in some bastardized fashion) they are god-givens

No dog in this fight. logically Some good features is not interchangeable with is complex

Re: Scala at scale at Databricks

#147

Post by valenterry »

StreamBright wrote: post #80
coffeecloud wrote: post #12

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

Same. Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem". I think F# is doing much better in this category. Scala can surprise you many unexpected ways. https://www.youtube.com/watch?v=v1wrWQcqLpo

F# has some nice features that scala is lacking, such as a more terse syntax for ADTs or type providers.

But other than that, Scala is miles ahead. And if you want to do pure FP, then Scala is the only option.

Also about that video, almost all of the points mentioned are either not available in other languages like F# anyways, or they already fixed in the latest Scala version.

Re: Scala at scale at Databricks

#148

Post by scalaIsKindaMeh »

MrPowers wrote: post #97
sireat wrote: post #90

Earlier quoted context omitted.

I seriously love Li Haoyi Scala libraries especially the JSON one. The hilarious thing is/was that Scala tutorials/books would show writing JSON parser as one examples. Yeah Scala makes it easy to write a JSON parser but you still do not want to roll your own. Case in point Odersky's Programming in Scala.

Yea, there was a JSON parser in the Scala standard lib at one point and it was then deprecated & removed. Guess it's easy to make a "hello world" JSON lib, but maybe not the best addition to the standard lib, haha.

JSON or XML?

Re: Scala at scale at Databricks

#149

Post by joelbluminator »

sireat wrote: post #87
joelbluminator wrote: post #77

Earlier quoted context omitted.

In what way is Scala a better Python? The 2 seem nothing alike to me.

Scala is almost as terse as Python for data munging/wrangling tasks (unlike say C#,C++ or Java) Scala is almost as much joy to write data munging tasks as Python (unlike say C#, C++, Java, and I have to say Golang). Chaining multiple maps and filters is so much more pleasurable than writing 4 nested loops with multiple ifs inside. The better part is that you can reliably deploy Scala unlike Python. We just had yet an…

Still. For some data scientist who knows nothing about programming, or for some CS student getting started with his first programming assignment - which is easier - Python or Scala? I'd say Python, by a lot.

Re: Scala at scale at Databricks

#150

Post by EdwardDiego »

lmm wrote: post #24
EdwardDiego wrote: post #21

Earlier quoted context omitted.

Why not just use Jackson? Sure, it doesn't have a clever DSL. But then, that's a feature.

Jackson has its own parallel "type registry" to make up for not being able to use something like Shapeless, because the problem still has to be solved. Getting a failure at compile time if there are any cases that wouldn't serialize properly is a huge advantage; a nontrivial proportion of production outages I've seen were Jackson.

I'll be honest, I had a quick look at Shapeless, and couldn't see how you'd apply to ensuring an entity serialized correctly.

But I'm keen to learn more :) I always prefer compile time to run time errors.

Post reply on HN