Hey all - I'm the release manager for Spark 1.1. Happy to answer any questions about Spark or this release.
Any plans to allow GraphX to work with Spark Streaming DStreams?
Announcing Apache Spark 1.1
11–15 of 15 posts
Re: Announcing Apache Spark 1.1
#12Hey all - I'm the release manager for Spark 1.1. Happy to answer any questions about Spark or this release.
Good news about the PySpark input format improvements. Does that also cover reading complex Parquet datatypes into SchemaRDDs with their native datatypes? When can we get a Databricks Cloud account (I'm already on the waiting list)?
Yeah, you can load Parquet data directly into SchemaRDD's in 1.1 and get the type conversion, including use of nested types. That's the long term solution for all of our storage integration is to go through the SchemaRDD API since it's a standard type description and we expect many data sources to integrate there.
Re: databricks cloud - shoot me an e-mail and I'll see if I can help. Right now demand exceeds supply for us on accounts, but I can try!
Re: Announcing Apache Spark 1.1
#13Great to see this: > This release adds significant internal changes to Spark focused on improving performance for large scale workloads. We looked at Spark Streaming briefly when choosing which CEP engine to use. We ended up not using it as its performance wasn't on par with other offerings. I hope the performance improvements they've done carry over to the spark streaming product. http://spark.apache.org/streaming/…
It is quite possible to build a unified storage and execution kernel that will allow you to simultaneously and seamlessly blend (1) streaming ingest path processing, (2) online indexing/storage to disk at wire speed, and (3) fast online query processing that immediately reflects both storage and ingest path. Saturating a 10 GbE connection with this kind of workload on an ordinary server is pretty simple if the system…
Everything above is a great summary of the CEP startup I was part of 7-8 years ago. It took us 3+ years to get the product off the ground to the stage where we started to see real sales. We've had to build custom code to handle networking, messaging, scheduling, storage, etc. to get to the required performance numbers.
Re: Announcing Apache Spark 1.1
#14Hey all - I'm the release manager for Spark 1.1. Happy to answer any questions about Spark or this release.
Any plans to allow GraphX to work with Spark Streaming DStreams?
You can call GraphX algorithms right now from within the Streaming API, for instance compute a graph on a windowed view of data.
Online graph algorithms aren't there yet (probably what you mean). We just started adding online MLlib algorithms, so this is the main focus for now.
Re: Announcing Apache Spark 1.1
#15Earlier quoted context omitted.
Good news about the PySpark input format improvements. Does that also cover reading complex Parquet datatypes into SchemaRDDs with their native datatypes? When can we get a Databricks Cloud account (I'm already on the waiting list)?
Don't the SchemaRDD already support Parquet? Although it'd be great if they supported CSVs.
There's work in progress to support importing CSV data as SchemaRDDs:
https://issues.apache.org/jira/browse/SPARK-2360 https://github.com/apache/spark/pull/1351