Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scala at scale at Databricks

databricks.com

Post reply on HN
Print view View original

101–110 of 174 posts

Re: Scala at scale at Databricks

#101

Post by jdmichal »

flowcont wrote: post #93
linspace wrote: post #45

Earlier quoted context omitted.

For most tasks programming it's more an engineering discipline than an intellectual pursuit. In that context complexity must always be justified. I know, and have been told, to have written hard to read code. In my case it's usually vectorized code in numpy or a C or cython extension for really hot code. But I always have a good reason, usually performance when it matters. It's usually said that premature optimizatio…

I'm stealing that phrase: "Premature abstraction is the root of all evil". It's hard for me sometimes to justify in a code review why an abstraction is not required (yet) when someone has put some effort into it.

I found this in another thread and it sounds like you might enjoy it also:

https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction

Re: Scala at scale at Databricks

#102

Post by jdmichal »

the_only_law wrote: post #55
misja111 wrote: post #54

Earlier quoted context omitted.

Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for. I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were thos…

> I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked…

I found this in another thread and it sounds like you might enjoy it also:

https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction

Re: Scala at scale at Databricks

#103

Post by ByteJockey »

AzzieElbab wrote: post #92
hyperbovine wrote: post #52

Earlier quoted context omitted.

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are wa…

I hear you. However the amount of category theory you need to understand 99.99% of FP is really really minimal. 10-15 pages of well written text

Y'know what, I'll bite. I can do 10-15 pages. You have a link?

Re: Scala at scale at Databricks

#104

Post by halfmatthalfcat »

mohanmcgeek wrote: post #14
coffeecloud wrote: post #12

Scala is an organizational red flag for me, especially if their recruiters are looking for "scala engineers."

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

Those are the least confusing libraries in the Scala ecosystem lol. Slick is just like any ORM-light framework. Akka-HTTP abstracts away all the Akka-specific things and you just create routes that return Futures essentially. How is spray-json complex? You're just using macros against your case classes to create (de)serializers.

If anything, you should be saying things like Shapeless provide the extra complexity.

Re: Scala at scale at Databricks

#105

Post by halfmatthalfcat »

derN3rd wrote: post #33

It amazes me that such big projects use monorepos. My company also though about doing so, because we have a few typescript projects that share a common module with business logic and everything lags behind when the common module was upgraded, but I don't know if I should sell them to switch over to a mono repo as well. My biggest pro argument would be the speed up in development and review time, because everything is…

SBT is actually amazing at maintaining a monorepo (or multi-project as they call it)

Re: Scala at scale at Databricks

#106

Post by Fiahil »

hocuspocus wrote: post #89
Fiahil wrote: post #78

Earlier quoted context omitted.

> But if you can, [...] Scala can be hugely productive and result in very reliable software. Yes, but, in my experience, the productivity gains are not that big. Strong developpers can and _will_ fare well in any technical environments, therefore choosing Scala over anything else won't give your team any advantage over a "simpler" technology. It's actually riskier to pick Scala, because you will increase your recruit…

> Save yourself, don't use Scala, pick Rust. That's an hilariously bad advice if you think hiring competent developers and onboarding juniors is hard in Scala. Most things companies do on the JVM, especially at scale, are going to be a lot harder in Rust and its ecosystem.

> That's an hilariously bad advice

Absolutely not! I'm serious, and I stand by it. Rust ecosystem is like a better Python. You have to consider why someone is willing to move away from Python and into a foreign language: Developer supply, performance, safety, ecosystem, political reasons.

I can't cover all cases so I'm going to focus on the one that makes the most sense _from my personal point of view_.

I do AI Software Enginering and Devops when I'm not project managing. So Python is the only and primary choice for all things related to Data Science, dashboards, APIs, pipelines and transformations (on data).

Why and where would I choose Rust ?

In data pipelines, for performances and memory efficiency. This also ensure we have proper boundaries and documentation (APIs, file formats, schemas) between Data Science-related code and pipelines.

In Dashboards and APIs, for improving type safety, and handling more concurrency on background computations than python without resorting to other frameworks (Celery, Airflow, ...).

In data versioning and supporting infrastructure, for additional type safety and mostly making sure that _nobody_ who is not a software engineer won't cross component boundaries by accident. These components are often very delicate and essentials for a working platform. Changing them can have ripples into other parts of the project, and I only want trained engineers on these.

In the opposite, direction, I could pick Python over Rust when it's most appropriate: Restricted developper availability, improving coordination with Data Scientists, single container deployments, existing experience with framework or ecosystem...

The main blocker for not using Rust isn't because the language is new or more complicated, it's because Python offers more advantages to a situation. At the difference from Scala, I can pick _any_ skilled developper and put him from Python to Rust without any side effects (appart from the training part). This is because writing Rust code is much more consistent at all skill level than Scala. There no "flavor" of Rust. There no bickering about "coding style". The Rust tooling is best in class (Scala's SBT is actually worst-in-class, I believe). You can use the same IDE in Rust an Python (VSCode). This leads to a very similar coding experience between writing type-hinted-Python and Rust. This is not the case between Python and Scala.

Re: Scala at scale at Databricks

#107

Post by mohanmcgeek »

lmm wrote: post #25
mohanmcgeek wrote: post #14

Earlier quoted context omitted.

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

The standard library is where modules go to die, just like in Python. And unlike Python, there's a good build/dependency manager available for Scala (Maven). Keep that stuff out of the core language, let it be libraries released on their own schedules.

JSON has a standardized spec and the API you design has to do a few things: Marshal and unmarshal.

How frequently do these need to change?

The problem with the library landscape is that one library will be promoted by the purists and another by the company with the money. Suddenly you have to learn both to understand Scala codebases.

Re: Scala at scale at Databricks

#108

Post by CraigJPerry »

sixminuteabs wrote: post #94

We develop the majority of our code in scala at my company (in fintech). There seems to be an experience in this thread that training in Scala is difficult. For anyone who is considering and wants another perspective - I have found that junior engineers tend to get up to speed more quickly in Scala than in other languages and that I can give them more freedom (because I know their programs won't compile if they don't…

>> I can give them more freedom (because I know their programs won't compile if they don't line their types up

In what meaningful way can you give them more freedom? When i think about how i spend my time with junior folks, it tends to be on things like:

1. Do they understand enough of our problem domain that i can begin to describe a problem that needs to be solved?

2. Have they understood the problem i described? How confident am i that they're not going to implement a solution to the wrong problem?

We don't spend time on "hey, you passed the parameters in the wrong order here".

To put it another way based on my experience, what's more useful to a junior developer?

Option A: Hey, here's a test case, here's how to run it. I need you to change the behaviour of this component to do X instead.

Or

Option B: Hey, here's a component with well designed types. I need you to change the behaviour of this component to do X instead.

My experience has been that Option A - play with the component hands on, try things, run it, tweak the data supplied to it and observe the results - these encourage engagement. The developer will be excited even.

And for Option B, they'll be quite dejected, not that invested in the project. I can't even compile the stupid thing because of some cryptic compile error.

Re: Scala at scale at Databricks

#109

Post by [deleted] »

_vertigo wrote: post #39
tlarkworthy wrote: post #31

Earlier quoted context omitted.

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

How did that conversation go? “We like your solution but did you seriously just use Maven?” What did they say exactly?

[deleted]

Re: Scala at scale at Databricks

#110

Post by mohanmcgeek »

EdwardDiego wrote: post #21
mohanmcgeek wrote: post #14

Earlier quoted context omitted.

It really should be. When you're trying to solve a business problem, Scala gets on the way. And when I say Scala, I mean libraries like akka http, spray-json and slick. I like FP parts of Scala overall. You would think that Scala 3 released in 2021 would have first class net/http and json that we can get rid of spray json. But no.

Why not just use Jackson? Sure, it doesn't have a clever DSL. But then, that's a feature.

These decisions aren't made by me.

can never convince the FP purists to let you use a non FP library.

Post reply on HN