Scala Data Pipelines for Music Recommendations at Spotify

My team has plans of moving away from Scala and towards Python primarily because of the job market. There is a feeling that it will be significantly easier to find a good data engineer who uses Python than it will be to find the same who uses Scala.

We're currently looking to hire a second engineer and have also been rethinking the direction of the data engineering team. Changing the language to Python was a suggestion for the hiring factor and because it is the popular language for data engineering. Scala has been a fun language over the last year and a half and I wouldn't mind sticking with it because it has been fun. For that reason, I have spent time comparing the two languages for data engineering and haven't come to a consensus on which one is ideal for my team.

The use of Scala predates me, it's been going on about 5 years although there's been a huge shift in that direction over the last year. This time last year I'd say it was about 50/50 Scala and Java and now almost all new code is written in Scala. I've been told that the initial "why" of scala was that it made it easy to write DSLs with which to write pipeline data flows w/o leaving the JVM. We have our own pipeline infrastructure written in house (in scala) although we're starting to move toward more OTS options (also in scala).

The surge over the last year has more to do with personnel changes within the group with more people joining who were more pro-Scala (such as myself), mainly for some combination of the functional trappings and the reduction of boilerplate and other nonsense from Java.

In terms of the job market, while we'd love to see a candidate who is a Scala wiz, we don't expect them to have an ounce of scala background. Personally I don't care if they have any java background either, although not everyone in my group agrees. I lean very heavily on the side of hiring smart people instead of hiring for buzzwords, assuming that we're not talking about contract positions.

FWIW the data scientists in our group are mostly split between R and Python, with some also writing a fair amount of scala. The latter has more to do with some of the in house pipeline management tools than any data science-y reasons.

It's not always easy to understand (from Odersky himself https://gist.github.com/odersky/6b7c0eb4731058803dfd#file-fo...):

def toVector: Vector[B] = fold(Vector[B]())(_ :+ _)(_ ++ _)

To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish.

I've also had to write code like this:

client.post(args).mapTo[Response].map(r => (r.success, r.serverException, r.unhandledException) match {

case (Some(response), None, None) => response

case (None, Some(serverEx), None) => throw serverEx

case (None, None, Some(unhandledEx)) => throw unhandledEx

})

simply because of the underlying api that other co-workers have built. When it takes you longer than 30 seconds to explain how a piece of code works to others, something is wrong.

All that being said, I do like the language; just not the compile times. It's also pretty close to ruby (with which I'm most familiar):

ruby:

numbers = [1, 2, 3, 4, 5]

numbers.select { |n| n >= 4 }

# [4, 5]

Scala:

val numbers = List(1, 2, 3, 4, 5)

numbers.filter(n => n >= 4)

// List(4, 5)