I'm on Spark Scala 2 project and I hate it. Basically any good Scala dev would never want to work on our ETL projects, so we get second rate Python or Java devs like me who bastardize the language to get anything to work. Most of our new stuff is all pyspark, hopefully we can replace Scala asap.
Scala 3 slowed us down?
101–110 of 195 posts
Re: Scala 3 slowed us down?
#102Earlier quoted context omitted.
Thank you for the clarification. If I understand correctly, these large expressions are created at compile-time, but the impact isn't felt until JIT occurs in the runtime environment. In that scenario, shouldn't the JIT just run once at startup, though? I'm still not quite understanding how JIT can take so much time in a production environment.
Because the jit will let the unoptimized code run a few (hundred) times to take measurements to know what needs to be optimized and how it needs to be optimized. This is a good solution and makes hotspot very effective. The problem is that it happens randomly a few minutes/seconds into the operation of the service. So you randomly have a big pause with the performance hit everytime you run the service. The upside is…
Ah, that’s interesting. I wasn’t aware that JIT-ing will do that sort of performance analysis first. Thank you for the explanation.
Re: Scala 3 slowed us down?
#103Earlier quoted context omitted.
> it's a language made by academics for academics to play with language design. It was a little weird it blew up in industry for a while. Yep. They have always been pretty honest about this. I think that it blew up in industry because it really was ahead of its time. Type systems were pretty uncool before Scala. It proved that you could get OO and FP in a single type system. Actually, a big part of reason for doing S…
> Type systems were pretty uncool before Scala I’m not up on programming language engineering as much as I should be at 37, could you elaborate a bit here? (To my untrained ear, it sounds like you’re saying Scala was one of the first languages that helped types break through? And I’m thinking that means, like, have int x = 42; or Foo y = new Foo()”
Not types, type-safety. Things like covariant and contravariant type declarations, implicit types (variables looked up by type instead of by label), and other things that you need to make a type safe system/service/application. The problem is that that feature of a language is massively oversold. Its nice but to pretend it prevents bugs or is even a great design goal is questionable and not backed up by research (as they claim).
Re: Scala 3 slowed us down?
#104The problem with Scala 3 is that nobody asked for it. The problem with Scala 2 is that the type inference part of the compiler is still broken. Nobody worked on that. Instead they changed the language in ways that don't address complaints. Completely ignore the market and deliver a product nobody wants. That's what happened here. PS Perhaps they should make an actual unit test suite for their compiler. Instead they h…
> The problem with Scala 2 is that the type inference part of the compiler is still broken. Nobody worked on that. Instead they changed the language in ways that don't address complaints.
Huh? Type inference is much more consistent and well-specified in 3. In 2 it was ad-hoc so and impossible to fix anything for one codebase without breaking another. There are plenty of legitimate complaints to be had about Scala 3, but this is absolutely not one of them.
Re: Scala 3 slowed us down?
#105> After upgrading the library, performance and CPU characteristics on Scala 3 became indistinguishable from Scala 2.13. Checking the bug mentioned, it was fixed in 2022. So, I’m wondering how one would upgrade to scala 3, while keeping old version of libraries? Keeping updated libraries is a good practice (even mandatory if you get audits like PCI-DSS). That part puzzled me more than the rest.
> So, I’m wondering how one would upgrade to scala 3, while keeping old version of libraries?
The normal way.
> Keeping updated libraries is a good practice
So is changing one thing at a time, especially when it's a major change like a language version upgrade.
Re: Scala 3 slowed us down?
#106The only issue I have with Scala 3 is Python envy, they should not have come up with a second syntax, and pushing it as the future. If anything is slowly down Scala 3 is that, including the tooling ecosystem that needs to be updated to deal with it.
Everything is up to date with the new syntax as far as I'm aware. Also, the compiler and scalafmt can rewrite one to the other. A project can pick whatever style it wants and have CI reformat code to that style.
> Everything is up to date with the new syntax as far as I'm aware.
The Eclipse plugin isn't, and none of the newer IDE integrations is reliable.
Re: Scala 3 slowed us down?
#107Earlier quoted context omitted.
Macro's are compile time, there is no runtime codegen. The problem was overly-frequent inlining generating enormous expressions, causing a lot JIT phase and slow execution.
That's not true, Spark's entire query engine relies on use of runtime codegen via macros/quasi quotes Look up the architecture of Catalyst + Tungsten https://www.databricks.com/glossary/catalyst-optimizer
Catalyst uses runtime codegen, sure, but the OP wasn't using that.
Re: Scala 3 slowed us down?
#108Earlier quoted context omitted.
> and a decade of tinkering has just fatigued everyone and destroyed any momentum the language once had. it's hard to buy it, considering that many of those "fatigued" moved on Kotlin, led by their managers' bs talking points.
Many of the Scala projects got people fired. Something the Scala devs largely ignore. Plus Scala support is truly awful even by the low standards of an OpenSource project. Then there is the fact that the Scala specific libraries are largely dead. Scala had/has a lot of promise. But how the language is marketed/managed/maintained really let a lot of people down and caused a lot of saltiness about it. And that is befor…
> Scala had/has a lot of promise. But how the language is marketed/managed/maintained really let a lot of people down and caused a lot of saltiness about it. And that is before we talk about the church of type-safety.
On the contrary, there was nothing wrong with Scala's marketing. What's damaged it is a decade of FUD and outright lies from the people marketing Kotlin.
Re: Scala 3 slowed us down?
#109I was involved in a Scala point version migration (2.x) migration a few years ago. I remember it being painful. Although I recall most of the pain was around having lots of dependencies and waiting for libraries to become available. At the time Scala was on upswing because it had Spark as its killer app. It would have been a good time for the Scala maintainers to switch modes - from using Scala as a testbed for inter…
Outside of Android work, has Kotlin really taken over? My understanding is that Java added a lot of functional programming and that took a lot of wind out of Scala's sails (though Scala's poor tooling certainly never helped anything).
Java did indeed add more FP to the language, but Java's type system is still fairly primitive compared to Scala's.
Re: Scala 3 slowed us down?
#110Earlier quoted context omitted.
Benchmarking requires a bit of different setup than the rest of the testing, especially if you want down to the ms timings. We have continous benchmarking of one of our tools, it's written in C++, and to get "same" results everytime we launch it on the same machine. This is far from ideal, but otherwise there be either noisy neighbours, pesky host (if it's vm), etc. etc. One idea that we thought was what if we can ru…
I agree for measuring latency differences you want similar setups. However, by running two versions of the app concurrently on the same machine they both get impacted more or less the same by noisy neighbours. Moreover, by inspecting the flamegraph you can, manually, see these large shifts of time allocation quickly. For automatic comparison you can of course use the raw data. In addition you can look at total cpu se…
Good points there - Thanks @spockz!