Scala at scale at Databricks • Hacker News Board

Re: Scala at scale at Databricks

#151

Post by lmm » Sat, Dec 11, 2021, 8:55 AM UTC

aweiland wrote: post #140

lmm wrote: post #50

Earlier quoted context omitted.

I don't think so. Kotlin is an unmaintainable soup of features: it can match all the selling-point examples of Scala but only by adding ad-hoc special cases for each one. (For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approa…

This is a poor straw man argument. Spring boot is an option, but so is Ktor, or Dropwizard. You are arguing you preference as fact.

I'm arguing my lived experience of working on Kotlin codebases. It's not about the specific framework, it's that people always use a framework that breaks the rules of the language, just like in Java, because the language itself isn't expressive enough. Whereas the thing that makes all those complex Scala libraries worthwhile is that they enable you to make a whole enterprise-scale codebase using just plain functions and values.

Re: Scala at scale at Databricks

#152

Post by mohanmcgeek » Sat, Dec 11, 2021, 9:03 AM UTC

halfmatthalfcat wrote: post #138

mohanmcgeek wrote: post #129

Earlier quoted context omitted.

Having used both Scala and Go for the past 6 years, I can tell that for the purposes of creating a database backed web API, they're definitely not at the same level of complexity. This isn't about the type system. This is about complicating things by making everything FP while getting nothing in return.

That's the great thing (and the point of many of the comments in this thread) about Scala - you don't have to do any FP with Scala if you so choose. You can operate in a completely OO fashion and gain all of the benefits of the language (primarily the type system) for free.

People promoting this view fail to understand that people work in teams and it only takes one FP zealot to force everyone to do FP and your PRs don't get merged unless it's idiomatic Scala.

Re: Scala at scale at Databricks

#153

Post by lmm » Sat, Dec 11, 2021, 9:03 AM UTC

EdwardDiego wrote: post #150

lmm wrote: post #24

Earlier quoted context omitted.

Jackson has its own parallel "type registry" to make up for not being able to use something like Shapeless, because the problem still has to be solved. Getting a failure at compile time if there are any cases that wouldn't serialize properly is a huge advantage; a nontrivial proportion of production outages I've seen were Jackson.

I'll be honest, I had a quick look at Shapeless, and couldn't see how you'd apply to ensuring an entity serialized correctly. But I'm keen to learn more :) I always prefer compile time to run time errors.

What Shapeless gives you is the ability to treat value-like types - case classes and sealed traits - generically. And so you can do "walk the object graph" style stuff at compile time - particularly, typeclass derivation.

https://circe.github.io/circe/codecs/auto-derivation.html is one popular library that does this - note that the `.asJson` will fail at compile time if you add a member to Greeting or Person whose type can't be serialized/deserialized (e.g. File or Lock or something) i.e. one that doesn't have an Encoder typeclass instance available.

Re: Scala at scale at Databricks

#154

Post by lmm » Sat, Dec 11, 2021, 9:05 AM UTC

mohanmcgeek wrote: post #107

lmm wrote: post #25

Earlier quoted context omitted.

The standard library is where modules go to die, just like in Python. And unlike Python, there's a good build/dependency manager available for Scala (Maven). Keep that stuff out of the core language, let it be libraries released on their own schedules.

JSON has a standardized spec and the API you design has to do a few things: Marshal and unmarshal. How frequently do these need to change? The problem with the library landscape is that one library will be promoted by the purists and another by the company with the money. Suddenly you have to learn both to understand Scala codebases.

Well literally today I got a notification from GitHub that I need to upgrade the version of jackson-databind that one of my project depends on because of a security vulnerability. And apparently that's the 8th such fix to this patch version of jackson-databind.

Re: Scala at scale at Databricks

#155

Post by lmm » Sat, Dec 11, 2021, 9:08 AM UTC

wiseowise wrote: post #57

lmm wrote: post #50

Earlier quoted context omitted.

I don't think so. Kotlin is an unmaintainable soup of features: it can match all the selling-point examples of Scala but only by adding ad-hoc special cases for each one. (For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approa…

> Kotlin is an unmaintainable soup of features Are you sure you're not confusing Kotlin with Scala? > For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approaches And that is a problem how? Stick to one style. > In practice Kotl…

> And that is a problem how? Stick to one style.

Switching an API from "a result or nothing" to "a result or an error message" happens all the time, and switching in the other direction is only slightly less frequent. And of course most programs have some APIs where one is appropriate and some where the other is. So consistency is valuable.

> https://github.com/spring-projects-experimental/spring-fu/tr...

Still reflection-based.

> There's nothing magical about it.

It's magical to anyone thinking in the language - it breaks the rules of the language, so you can't reason about what it does.

Re: Scala at scale at Databricks

#156

Post by esarbe » Sat, Dec 11, 2021, 11:02 AM UTC

burntoutfire wrote: post #73

amir734jj wrote: post #23

Earlier quoted context omitted.

In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :) Don't be negative about technology. Be positive.

Scala + Spark is absolutely the sane face of Scala. Try using stuff like cats (basically Haskel ported to Scala) to experience the deep end of the pool.

To discard Cats as 'basically Haskell ported to Scala' is doing it a bit of a disservice. Yes, it explores pure functional programming, but that design space is huge.

Re: Scala at scale at Databricks

#157

Post by esarbe » Sat, Dec 11, 2021, 11:19 AM UTC

Scala is an amazing language I love writing software in. The tools it offers to model business domains, the power of expressive types make it next-to-none when it comes to shape, remodel and refactor code.

Then there's the ecosystem. Sure, you have the whole of the Java ecosystem, but then there are also tons and tons of Scala libraries that are so well designed (Quick shout-out to Lihaoyi here; your libraries are amazing!) and work together so well is astounding.

I was very worried with the migration path to Scala 3 but that turned to be out pretty much painless. Scala-2.13-backward-compatiblity is amazing and together with scalafix the transition is as smooth as anyone could reasonably expect.

Anyone getting into Scala or worried about complexity with Scala should have a good look at https://www.handsonscala.com/

Re: Scala at scale at Databricks

#158

Post by esarbe » Sat, Dec 11, 2021, 11:23 AM UTC

jvican wrote: post #122

Fiahil wrote: post #106

Earlier quoted context omitted.

> That's an hilariously bad advice Absolutely not! I'm serious, and I stand by it. Rust ecosystem is like a better Python. You have to consider why someone is willing to move away from Python and into a foreign language: Developer supply, performance, safety, ecosystem, political reasons. I can't cover all cases so I'm going to focus on the one that makes the most sense _from my personal point of view_. I do AI Softw…

You can use Scala with VS Code just in the same way you can do it with Rust. Tooling support for Scala is world-class in many ways. I think your knowledge is pretty outdated.

Seconded.

While the criticism that Scala's tooling support might have been true some six or seven years ago, nowadays it is top notch.

Re: Scala at scale at Databricks

#159

Post by esarbe » Sat, Dec 11, 2021, 11:37 AM UTC

StreamBright wrote: post #80

coffeecloud wrote: post #12

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

Same. Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem". I think F# is doing much better in this category. Scala can surprise you many unexpected ways. https://www.youtube.com/watch?v=v1wrWQcqLpo

> Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem".

Could you elaborate? Because that's not been my experience at all. I have found Scala to provide many tools to wrap around the idiosyncrasies of the underlying imperative that they are barely noticeable anymore.

Re: Scala at scale at Databricks

#160

Post by cmollis » Sat, Dec 11, 2021, 11:53 AM UTC

MrPowers wrote: post #35

Scala is a great language when Li Haoyi ecosystem of libraries is used. His libs do crazy Scala stuff under the hood, but expose a clean end user interface. The problem with Scala is that most ppl don't use his libs. Here's a blog post I wrote on reading / writing JSON using one of Li's libs: https://mungingdata.com/scala/read-write-json/ There's now a completely sensible option for JSON in Scala. But there are also…

agree.. I really like scala and Li's blog and book have helped me immensely. I use it mostly for spark, but I've found it really useful now for other things. The concurrency really is pretty nice (compared to other languages). His API's are written in the same way as his blog and his book: zero condescension; he's not constantly trying to impress you. This seems to be the problem with the language.. the language allows you to really go nuts if you want to and the result can be an incomprehensible type system that takes a week of hacking just to figure out what it wants you to do. That said, json4s isn't terrible.