Scala Data Pipelines for Music Recommendations at Spotify

Post by lmm » Tue, Jan 13, 2015, 11:00 AM UTC

stoplight wrote: post #10

lmm wrote: post #3

Earlier quoted context omitted.

There are certainly some cool big-data tools in Scala (personally I find Spark much more compelling than Scalding, but the fact that I'm even comparing two reasonable choices is more than you'd get with many languages), but I don't think that's what's driving adoption. My case for Scala would be that it's a great language to write and an even better one to read: even more expressive than Python but much safer to refa…

> and an even better one to read...and it's easy to understand which parts of the code do what It's not always easy to understand (from Odersky himself https://gist.github.com/odersky/6b7c0eb4731058803dfd#file-fo... ): def toVector: Vector[B] = fold(Vector[B]())(_ :+ _)(_ ++ _) To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish. I've a…

> def toVector: Vector[B] = fold(Vector[B]())(_ :+ _)(_ ++ _)

> To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish.

Compare what the same code would look like in Ruby; something like (hope I get the syntax right):

    def toVector = fold(new Vector()) {|x, y| x :+ y} {|x, y| x ++ y}

Are the extra |x, y|s actually clarifying anything? Or are they just syntactic ceremony? Maybe it's just my scala experience talking, but I think the scala example is clearer; there's very little performance to get in the way, just the meat of what the function's actually doing

(You can think that's a good or bad function to have, but that's a library question, not a language question).

> simply because of the underlying api that other co-workers have built.

You can write bad APIs in any language; with a more sensible one that would look like:

    client.post(args).map{
      case Success(response) => response
      case ServerError(ex) => throw ex
      case UnhandledError(ex) => throw ex
    }

Explicitly handling the different cases is exactly the kind of debugging advantage I was talking about; it takes up-front effort to distinguish between ServerError and UnhandledError, but the result is code where you can see all the possibilities and know exactly where any given failure might be happening. (And again, the language doesn't force you to do this; you can just write the happy path and allow any kind of exception to happen at any point. But you'll pay the price in debugging, as I have in Python, and as I presume you have in Ruby).

Re: Scala Data Pipelines for Music Recommendations at Spotify

Re: Scala Data Pipelines for Music Recommendations at Spotify