Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Towards Scala 3

scala-lang.org

Post reply on HN
Print view View original

261–270 of 383 posts

Re: Towards Scala 3

#261

Post by RBerenguel »

sandGorgon wrote: post #256
diogofranco wrote: post #212

Earlier quoted context omitted.

pyspark might be the go-to language for data scientists playing with the spark repl, or MLLib, but for production data engineering, scala is still king. Besides performance and the obvious fact that not knowing scala makes it difficult to understand the underlying Spark code, there are multiple ways in which scala is more natural to develop in (many libraries are for scala only, for example).

I don't think so. Python and data frames is arguably more natural to think about and reason than scala. I have no doubt that scala is more performant and the "fat" jar mechanism makes dependency management and codeshipping very easy (it's still tricky to install python dependencies on your spark nodes), but the pandas ecosystem is definitely more intuitive to understand.

I have the impression you are leaning towards thinking of data analytics (pandas, data frames, etc) whereas I and some other commenters may be thinking more of more data pipelining kind of architectures, where you can't afford wrong typing, scale is quite large and you are not even doing the kind of operations pandas dataframes are useful for

Re: Towards Scala 3

#262

Post by RBerenguel »

sandGorgon wrote: post #259
kprybol wrote: post #204

Earlier quoted context omitted.

It depends on the use case. Our work primarily revolves around extending Spark with custom pipelines, models, ensembles, etc. to be deployed into our production systems (petabyte scale). Scala was really the only way to go for us.

I can understand performance difference, but I have not generally seen a difference in building custom pipelines and ensembles .. although I grant I'm not at your scale yet. What kind of specific pipelines did you have trouble in pyspark ?

Although we decided to start using Scala specifically because PySpark was not as performant (2.0 is not so far ago), a reasonable use case I keep always in mind is aggregation (and in general any API which is still not solid/experimental/under work). Python bindings are always the last to be available (because all groundwork is being done in Scala). We have a relatively large scale process that takes advantage of custom-built aggregation methods on top of groupedDatasets, where we can pack a good deal of logic in the merge and reduce steps of aggregation. We could replicate this in Python using reducers, but aggregating makes more sense semantically, which makes the code is easier to understand. Also, the testing facilities for Spark code under Scala are a bit more advanced than under Python (they are not super-great, but are better), even without considering that being strongly typed makes a whole kind of errors impossible, right out of the compiler.

I very, very rarely think of using PySpark (and I have way more experience with Python than with Scala) when working with Spark. In a kitchen setting, it would be like having to prepare a cake and having to choose between a fork and a whisker. I can get it done with the fork, but I'll do a better and faster job with the whisker.

Re: Towards Scala 3

#263

Post by lmm »

bad_user wrote: post #207
lmm wrote: post #83

Earlier quoted context omitted.

> It's obvious that Scala was specifically designed to be a solid foundation on which one can implement the concepts of a little known branch of mathematics called the category theory, and almost every design choice in the language flows from there. Nonsense. Scala was designed to make programming easier and safer than Java - XML literals and pattern matching certainly have nothing to do with category theory. Scala's…

> Scala's for/yield has surprising behaviour ... because it doesn't have a categorical grounding Not sure what you're talking about, but for/yield is syntactic sugar for map/flatMap, just like the "do notation" is in Haskell. Of course it has theoretical grounding, because it wouldn't work without flatMap being the monadic bind. > mixing lists and sets Again, not entirely sure what you're talking about, but if true,…

I mean if you write something like:

    (for {
      x
the answer will be rather surprising, because in Scala for/yield is not necessarily monadic.

Re: Towards Scala 3

#264

Post by jrq »

braindongle wrote: post #260
jrq wrote: post #122

Disclaimer : I'm nobody important, but I do have an opinion If scala team were to disavow sbt, that'd be the single best thing they could possibly do for the ecosystem. I used to write a lot of scala, and working with sbt was enough to eventually get under my skin. I really like some of the OOP aspects of scala, the left-to-right style of thinking matches how my brain works. Scala having a lot to offer can distract n…

Please don't use "guys" this way. This is not PC nitpicking, It's important. "People" or "folks" may not have the ring you're after, but too bad.

Some of my biggest stressors in the past came from trying to change people or believing that I was entitled to have people act and react the way I wanted them too.

I don't give that privilege to others and I don't expect anybody else to.

I say guys because my developers and analysts and quality managers are all men. The only women I work with are two secretaries, and they're not even in my department.

What's too bad, friend, is that you have deluded yourself into believing you have the right of way over another person, for any reason whatsoever.

Bless your heart!

Re: Towards Scala 3

#265

Post by Cyph0n »

AsyncAwait wrote: post #252
Cyph0n wrote: post #235

Earlier quoted context omitted.

> CLion is the only IDE where they have tie ins to GDB etc. so the Native plugin is only for that Both me and the parent were referring to Native. So they are charging for a Kotlin Native debugger, no? Do they provide a GDB fork or something that can use without CLion?

> Both me and the parent were referring to Native. I know. So they are charging for a Kotlin Native debugger, no? No, Kotlin Native can interface with GDB or LLDB, they're charging for CLion which provides a GUI interface atop of these. > Do they provide a GDB fork or something that can use without CLion? No need to fork, you just won't have the frontend that Clion provides.

> I know.

So why did you refer to Kotlin on the JVM?

> No, Kotlin Native can interface with GDB or LLDB, they're charging for CLion which provides a GUI interface atop of these.

Cool, thanks for the response.

Re: Towards Scala 3

#267

Post by bitcharmer »

braindongle wrote: post #260
jrq wrote: post #122

Disclaimer : I'm nobody important, but I do have an opinion If scala team were to disavow sbt, that'd be the single best thing they could possibly do for the ecosystem. I used to write a lot of scala, and working with sbt was enough to eventually get under my skin. I really like some of the OOP aspects of scala, the left-to-right style of thinking matches how my brain works. Scala having a lot to offer can distract n…

Please don't use "guys" this way. This is not PC nitpicking, It's important. "People" or "folks" may not have the ring you're after, but too bad.

How is this not PC nitpicking. You really need a special kind of mindset to interpret completely harmless and generic term in this way.

Please take your SJW-ism somewhere else. This is HN.

Re: Towards Scala 3

#268

Post by amckinlay »

Last time I tried Scala I had to give up because I couldn't understand what a "trampoline" was, or how to use it to get multi-function tail recursion to work. Probably not Scala's fault.

Re: Towards Scala 3

#269

Post by eweise »

stusmall wrote: post #172
Bizarro wrote: post #26

Earlier quoted context omitted.

I don't think many companies are betting on Scala anymore. F# is just as irrelevant in the grand scheme of things. Let's face it, Kotlin hurts Scala adoption.

Kotlin and Java 8. I started using Scala in Java 7 era and loved it. When Java was stagnating it seemed like an alternate language that targeted the JVM was the best option. Progress with the language has really picked up a lot. There is a lot that Scala has that Java doesn't, but now its not enough to make it worth it. Java is so much easier to work with on a large team.

The two languages though also have preferred libraries / frameworks. I code in Java again and the main thing that bugs me is being always in the Spring ecosystem. There is a mindset in javaland that every technology needs to be wrapped by Spring. Scala gave me the freedom to choose alternative libraries.

Re: Towards Scala 3

#270

Post by lmm »

MichaelMoser123 wrote: post #218
lmm wrote: post #192

Earlier quoted context omitted.

The reality is that a language lives or dies by its ecosystem - particularly when it comes to a language like Scala that's in a tightly symbiotic relationship with its IDEs (the next time someone tries to sell you a "visual programming language", look at Scala for a language that makes really effective use of the GUI for programming without compromising the things that make textual programming languages good - see e.…

> if the language is to live, I have to convince other people it's the best Aren't JVM languages interoperable - because they use the same classfile/byte code format? What does it matter if the jar is written in java or scala, as long as it is possible to use the external interface from any of these languages?

> Aren't JVM languages interoperable - because they use the same classfile/byte code format?

Up to a point - libraries from another paradigm tend not to be idiomatic, but I could live with that. I'm more worried about tooling availability - I talked about IDE support in particular, and it's also things like profilers and monitoring/instrumentation.

Post reply on HN