Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

31–40 of 174 posts

Re: Scala at scale at Databricks

#31

Post by tlarkworthy »

philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

Re: Scala at scale at Databricks

#32

Post by wiseowise »

wiradikusuma wrote: post #9

I completely agree with their pragmatic approach to Scala. Scala is a very powerful language, and you can do super crazy stuff. My approach to writing Scala code is similar: 1. Write the code like a better-Java/typesafe-Python (that is, imperative style). 2. Glance through the code. For collections, usually there's a more concise approach. 3. Read the code again. If you think what you did in #2 is making the code _to…

> that is, imperative style

Yikes. Just switch to Java already.

Re: Scala at scale at Databricks

#33

Post by derN3rd »

It amazes me that such big projects use monorepos.

My company also though about doing so, because we have a few typescript projects that share a common module with business logic and everything lags behind when the common module was upgraded, but I don't know if I should sell them to switch over to a mono repo as well. My biggest pro argument would be the speed up in development and review time, because everything is in the same PR/MR and not split into each project Are there any huge downsides?

Re: Scala at scale at Databricks

#34

Post by the_only_law »

ackfoobar wrote: post #29
jongorer wrote: post #28

Earlier quoted context omitted.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage. Alternatively you can work in Go where braindead simplicity is the mandated norm.

I sympathize with your thinking. There are two "readabilities". 1. How easy it is for an average programmer to understand the code. 2. How understandable the code actually is if the reader is fluent with that prior knowledge. If one finds the latter is much better than code written in conventional techniques, but people are scared off by the former, it can get pretty frustrating, and "people are stupid" may be a conc…

So, I’ll probably get shit on for this, but I really cannot wait to leave this industry.

I hate how “best tool for the job” really just means “what we can have a large hiring pool of people who’ve only ever learned one programming style and possibly only one language of” or “for legacy reasons this is the only choice”.

Yes I’m familiar with reality, I just hate it.

Re: Scala at scale at Databricks

#35

Post by MrPowers »

Scala is a great language when Li Haoyi ecosystem of libraries is used. His libs do crazy Scala stuff under the hood, but expose a clean end user interface. The problem with Scala is that most ppl don't use his libs.

Here's a blog post I wrote on reading / writing JSON using one of Li's libs: https://mungingdata.com/scala/read-write-json/ There's now a completely sensible option for JSON in Scala.

But there are also 16 other Scala JSON lib options: https://stackoverflow.com/questions/8054018/what-json-librar...

It's so annoying working on a Scala project with dependencies that all use different JSON libs.

Even new, awesome projects like Delta Lake still use Scala dependencies like Scalatest and SBT that should be replaced with Li's libs in my opinion (utest & Mill).

Scala could be an amazing ecosystem if the community unified around Li's libs. It just doesn't seem like that's going to happen. Li wrote the "whats wrong with SBT" article back in 2017 and it seems like everyone is still using SBT.

I wrote an article on the maintenance challenges with Scala projects that trended on HN a while back: https://mungingdata.com/scala/maintenance-nightmare-upgrade/. Li's libs would solve a lot of these issues cause most of them are intentionally dependency free.

It was frustrating for me to see how Li basically singlehandedly fixed the Scala library ecosystem, but how none of the projects I actually used depended on his libs. I ended up leaving Scala after several years even though I like the language when it's used a certain way.

Scala is such a powerful programming language that has so much potential to be mainstream, but it's just so abused by "genius programmers".

Re: Scala at scale at Databricks

#36

Post by lmm »

ackfoobar wrote: post #27
bamboozled wrote: post #22

Earlier quoted context omitted.

I worked at a company full of Scala zealots and elites. It was insanely unproductive and people spent a lot of time trying to design around all the floors in Akka, trying to get Akka working in modern cloud computing environments and the like. I think at some stage specialists from some Akka pro shop had to be air dropped in to try fix it all, but it didn't help much. It always made me laugh how well all the services…

My bad experience with Scala has more to do with the FP side (cats and all that). Akka has been quite pleasant to me, so I'm quite surprised by this. Is the business problem not suited to the actor model in the first place?

I've never seen a case where the actor model actually helped with the business problem. Every time I saw Akka used, replacing it with straightforward Futures-and-for/yield code was an improvement.

Cats et al are frequently abused - 90% of the time rather than using a complex tool to solve a complex problem you want to think about it a bit and simplify it. But when you need them they do things that you can't do any other way, and they're vastly better than the reflection/AOP/bytecode-manipulation that people use to solve the same problem in Java or Kotlin.

Re: Scala at scale at Databricks

#37

Post by the_only_law »

lmm wrote: post #36
ackfoobar wrote: post #27

Earlier quoted context omitted.

My bad experience with Scala has more to do with the FP side (cats and all that). Akka has been quite pleasant to me, so I'm quite surprised by this. Is the business problem not suited to the actor model in the first place?

I've never seen a case where the actor model actually helped with the business problem. Every time I saw Akka used, replacing it with straightforward Futures-and-for/yield code was an improvement. Cats et al are frequently abused - 90% of the time rather than using a complex tool to solve a complex problem you want to think about it a bit and simplify it. But when you need them they do things that you can't do any ot…

Not using Akka, but I have a side project right now being done primarily in Erlang that felt natural to architect with Erlang “actor model” and the OTP (not the only reason I chose it of course, Erlang’s history and certain other features from that history made it a natural fit). There’s a small JVM-integrated component as well, but I’ll likely just be developing it in Java as it’s really just involved implementing an old JSR and I don’t foresee more advanced feature coming too much in handy.

Can’t speak to category theory. Not familiar with it, but I have done my fair share of “reflection metaprogramming” that might get me me shot (dynamically constricting and using generic types through reflection gets pretty nasty).

Re: Scala at scale at Databricks

#38

Post by wiseowise »

ackfoobar wrote: post #26
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

> who both understands those advanced concepts and is not a FP fanatic.

Oxymoron.

Re: Scala at scale at Databricks

#39

Post by _vertigo »

tlarkworthy wrote: post #31
philipkglass wrote: post #3

I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them. I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing l…

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

How did that conversation go?

“We like your solution but did you seriously just use Maven?”

What did they say exactly?

Re: Scala at scale at Databricks

#40

Post by wiseowise »

the_only_law wrote: post #34
ackfoobar wrote: post #29

Earlier quoted context omitted.

I sympathize with your thinking. There are two "readabilities". 1. How easy it is for an average programmer to understand the code. 2. How understandable the code actually is if the reader is fluent with that prior knowledge. If one finds the latter is much better than code written in conventional techniques, but people are scared off by the former, it can get pretty frustrating, and "people are stupid" may be a conc…

So, I’ll probably get shit on for this, but I really cannot wait to leave this industry. I hate how “best tool for the job” really just means “what we can have a large hiring pool of people who’ve only ever learned one programming style and possibly only one language of” or “for legacy reasons this is the only choice”. Yes I’m familiar with reality, I just hate it.

Apparently "best tool for the job" means find he bottom of the barrel that can still engage somehow with the codebase. This Idiocracy went to the point where returning function pointers gets questioned during code review because "this might be too complex compared to imperative call".
Post reply on HN