Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

.Net for Apache Spark Preview

devblogs.microsoft.com

Post reply on HN
Print view View original

1–10 of 24 posts

Re: .Net for Apache Spark Preview

#4

Post by aeroevan »

imbac82 wrote: post #3
kenhwang wrote: post #2

What's probably more interesting is how similar .net, scala, and python are in query performance. Not sure if that can be attributed to great python performance, or really bad scala/.net performance.

https://devblogs.microsoft.com/dotnet/introducing-net-for-ap...

Python 2.7? Also, the new Apache Arrow integration changes the python performance characteristics a lot; I wonder if they are using arrow for their JVM CLR interop, if not that probably would be a good idea.

Re: .Net for Apache Spark Preview

#5

Post by aeroevan »

kenhwang wrote: post #2

What's probably more interesting is how similar .net, scala, and python are in query performance. Not sure if that can be attributed to great python performance, or really bad scala/.net performance.

Most of PySpark is simply telling the JVM what to do, it's not actually running python directly. UDFs are where the real differences are, and they mentioned CLR UDFs serialize the spark Rows 2x faster than Python, but it's not clear if they were using apache arrow enabled pandas UDFs which are 3x-100x faster:

https://databricks.com/blog/2017/10/30/introducing-vectorize...

Re: .Net for Apache Spark Preview

#6

Post by polskibus »

aeroevan wrote: post #4
imbac82 wrote: post #3

Earlier quoted context omitted.

https://devblogs.microsoft.com/dotnet/introducing-net-for-ap...

Python 2.7? Also, the new Apache Arrow integration changes the python performance characteristics a lot; I wonder if they are using arrow for their JVM CLR interop, if not that probably would be a good idea.

It would be interesting to hear about it from MS. Do you know of other settings / configurations / features that could greatly influence the result of such comparison?

Re: .Net for Apache Spark Preview

#7

Post by flowerlad »

Python is a second class citizen in the world of Spark. Perf issues for UDFs. Some functions are only available through Scala. Python support for new features is always late. And so on. It is good to have .NET support but I will stick with Scala for the same reasons I switched from Python to Scala.

Re: .Net for Apache Spark Preview

#8

Post by polskibus »

aeroevan wrote: post #4
imbac82 wrote: post #3

Earlier quoted context omitted.

https://devblogs.microsoft.com/dotnet/introducing-net-for-ap...

Python 2.7? Also, the new Apache Arrow integration changes the python performance characteristics a lot; I wonder if they are using arrow for their JVM CLR interop, if not that probably would be a good idea.

Your post made me curious and I raised the issue with MS at https://github.com/dotnet/spark/issues/45. I hope that benefits the community and gets MS on the right track (by finally supporting Arrow).

Re: .Net for Apache Spark Preview

#9

Post by pjmlp »

flowerlad wrote: post #7

Python is a second class citizen in the world of Spark. Perf issues for UDFs. Some functions are only available through Scala. Python support for new features is always late. And so on. It is good to have .NET support but I will stick with Scala for the same reasons I switched from Python to Scala.

That is the exact reason that I learned to stay with platform languages for production code, even if there are more interesting ones trying to plug into it.

FFI, extra debugging layers and lesser tooling integration don't pay off a couple of language feature bullet points.

Re: .Net for Apache Spark Preview

#10

Post by vvladymyrov »

I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net).

But performance is not the only thing - there is also ability to debug issues. For this you still need to dig into Apache core which is in Scala.

This implementation in .Net would be "gateway drug" for moving your production to Scala/JVM.

It happened to me with PySpark - majority or tasks at hand can be solved with PySpark. But digging into the issues and stack traces brought me to Scala internals of Apache Spark. As a result in cases when python specific libraries are not needed and high performance needed I would write Spark programs Scala from the beginning.

Post reply on HN