[deleted]
.Net for Apache Spark Preview
21–24 of 24 posts
Re: .Net for Apache Spark Preview
#22Earlier quoted context omitted.
These are all good points. Debugability and general support for the development lifecycle are important. We are definitively working on providing first class development experiences for .NET developers. .NET for Apache Spark is already available as a nuget package for local install. We are currently working on adding support to VS Code, Visualstudio etc. Feel free to provide us your preferred dev platform. [Disclaime…
Thanks for the response! FWIW, I was speaking specifically to being able to run Spark, and manage its lifecycle, all inside the same process as the unit test code. Which is something that I'll openly concede isn't much more than a fun party trick for most people's purposes, but it does happen to serve me well. In a past life, I was involved in data engineering at a .NET shop, and being able to migrate parts of our pr…
Thanks... More idiomatic F# support is on the roadmap
Re: .Net for Apache Spark Preview
#23I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net). But performance is not the only thing - there is also ability to debug i…
Same, I recently moved an ML pipeline from PySpark to pure Python because of the debug-ability issue. The data science team, who managed the project, were experts in Python but relatively weak in Scala/Java. There were many issues were an improper data type may blow up pickling in the Java side and return absolutely cryptic errors. It was also difficult to do any sort of integration test and profiling on the code - t…
Indeed the real sad part is you can’t lead teams there early (premature optimization). Everybody seems to make the same rough transition on their own.
Re: .Net for Apache Spark Preview
#24What's probably more interesting is how similar .net, scala, and python are in query performance. Not sure if that can be attributed to great python performance, or really bad scala/.net performance.
A large percentage of Spark code is really just assembling lego blocks. The built-in blocks are themselves all written in Java or Scala, and the performance of the code that stacks them together is negligible.
It's mainly when you start writing custom UDFs (IOW, fabricating your own lego blocks) that platform interop and the performance of your language of choice become a big deal.