Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

.Net for Apache Spark Preview

devblogs.microsoft.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: .Net for Apache Spark Preview

#11

Post by bunderbunder »

vvladymyrov wrote: post #10

I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net). But performance is not the only thing - there is also ability to debug i…

On a somewhat related note, for my purposes the real deciding factor in sticking with Scala/JVM for (production) Spark work is testability: With that setup, it's dead easy to fire up a local Spark context, run unit tests against it, and keep the tests running reasonably fast.

Re: .Net for Apache Spark Preview

#12

Post by vvladymyrov »

bunderbunder wrote: post #11
vvladymyrov wrote: post #10

I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net). But performance is not the only thing - there is also ability to debug i…

On a somewhat related note, for my purposes the real deciding factor in sticking with Scala/JVM for (production) Spark work is testability: With that setup, it's dead easy to fire up a local Spark context, run unit tests against it, and keep the tests running reasonably fast.

For python guys - pyspark is now also installable from pip as a package (it include some .jars so it is ~100Mb size python package). So my team for local unit tests installs pyspark as a package.

Re: .Net for Apache Spark Preview

#13

Post by MichaelRys »

bunderbunder wrote: post #11
vvladymyrov wrote: post #10

I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net). But performance is not the only thing - there is also ability to debug i…

On a somewhat related note, for my purposes the real deciding factor in sticking with Scala/JVM for (production) Spark work is testability: With that setup, it's dead easy to fire up a local Spark context, run unit tests against it, and keep the tests running reasonably fast.

These are all good points. Debugability and general support for the development lifecycle are important. We are definitively working on providing first class development experiences for .NET developers. .NET for Apache Spark is already available as a nuget package for local install. We are currently working on adding support to VS Code, Visualstudio etc. Feel free to provide us your preferred dev platform. [Disclaimer: I am Program Manager for the .NET for Apache Spark effort]

Re: .Net for Apache Spark Preview

#14

Post by MichaelRys »

polskibus wrote: post #6
aeroevan wrote: post #4

Earlier quoted context omitted.

Python 2.7? Also, the new Apache Arrow integration changes the python performance characteristics a lot; I wonder if they are using arrow for their JVM CLR interop, if not that probably would be a good idea.

It would be interesting to hear about it from MS. Do you know of other settings / configurations / features that could greatly influence the result of such comparison?

Since we started replying to these points on the Github thread at https://github.com/dotnet/spark/issues/45, I am suggesting to continue the discussion there. As mentioned there, we want to be transparent with the benchmark code and systems we use. We are currently working on arrow support to compare fairly.

Re: .Net for Apache Spark Preview

#15

Post by nemothekid »

vvladymyrov wrote: post #10

I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net). But performance is not the only thing - there is also ability to debug i…

Same, I recently moved an ML pipeline from PySpark to pure Python because of the debug-ability issue. The data science team, who managed the project, were experts in Python but relatively weak in Scala/Java. There were many issues were an improper data type may blow up pickling in the Java side and return absolutely cryptic errors. It was also difficult to do any sort of integration test and profiling on the code - the start of moving off of Spark originally started as a way to do integration testing and profiling.

Re: .Net for Apache Spark Preview

#17

Post by manigandham »

tombert wrote: post #16

How is this different than Mobius [1]? [1] https://github.com/Microsoft/Mobius

From the github repo: https://github.com/dotnet/spark#inspiration-and-special-than...

>> Mobius: C# and F# language binding and extensions to Apache Spark, a pre-cursor project to .NET for Apache Spark from the same Microsoft group.

Re: .Net for Apache Spark Preview

#18

Post by jongalloway2 »

tombert wrote: post #16

How is this different than Mobius [1]? [1] https://github.com/Microsoft/Mobius

Here's my understanding:

- Mobius is .NET Framework / Mono based and x-plat isn’t great, .NET for Apache Spark is .NET Core / .NET Standard and built with x-plat as a primary concern

- Mobius only targets up to Spark 2.0; while Spark LTS is up to 2.4 now

- .NET for Apache Spark is built to take advantage of .NET Core performance improvements, showing big advantages over Python and R bindings, especially when user defined functions are a major factor

- .NET for Apache Spark is driven by lessons learned and customer demand, including major big data users inside and outside Microsoft

Disclaimer: I know people that worked on this and helped from .NET Foundation side, but the above is my non-official summary from readme's and stuff.

Re: .Net for Apache Spark Preview

#19

Post by bunderbunder »

MichaelRys wrote: post #13
bunderbunder wrote: post #11

Earlier quoted context omitted.

On a somewhat related note, for my purposes the real deciding factor in sticking with Scala/JVM for (production) Spark work is testability: With that setup, it's dead easy to fire up a local Spark context, run unit tests against it, and keep the tests running reasonably fast.

These are all good points. Debugability and general support for the development lifecycle are important. We are definitively working on providing first class development experiences for .NET developers. .NET for Apache Spark is already available as a nuget package for local install. We are currently working on adding support to VS Code, Visualstudio etc. Feel free to provide us your preferred dev platform. [Disclaime…

Thanks for the response!

FWIW, I was speaking specifically to being able to run Spark, and manage its lifecycle, all inside the same process as the unit test code. Which is something that I'll openly concede isn't much more than a fun party trick for most people's purposes, but it does happen to serve me well.

In a past life, I was involved in data engineering at a .NET shop, and being able to migrate parts of our process to something like Spark without having to rewrite or otherwise severely damage it would have made me very happy. Even better if I could stay inside Visual Studio, and bang on it from an F# interactive session.

Wild speculation, but if you can produce type providers that know how to tame `DataSet[Row]`, you might have some nonzero number of F# hipsters like me kissing your feet.

(Or not. Like I said, my perspective on Spark is unusual.)

Post reply on HN