Semi-related but does anyone know if zeppelin supports java yet?
Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
11–20 of 25 posts
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#12Earlier quoted context omitted.
I really wish there were more tutorials like this on how to set up Spark and other Big Data tools (TensorFlow) on cloud computing, as that personally has been the primary barrier to starting work with extra large amounts of data. Just as an FYI, we[1] are working on an open source, cloud based Machine Learning / Big Data platform that might be of interest to you. It's not all ready yet, but when it is, there will be…
Disclosure: I work at Google on Kubernetes. If you'd like to do this in containers/Kubernetes, we'd love to highlight your work! Kubernetes runs great on AWS (as well as GCP, Azure and elsewhere), so no cloud migration required.
Shameless plug:
Pachyderm is another way to run big data workloads in Kubernetes. github.com/pachyderm/pachyderm. We spoke at KubeCon SF last year and v1.0 is coming out next month!
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#13Earlier quoted context omitted.
Disclosure: I work at Google on Kubernetes. If you'd like to do this in containers/Kubernetes, we'd love to highlight your work! Kubernetes runs great on AWS (as well as GCP, Azure and elsewhere), so no cloud migration required.
Shameless plug: Pachyderm is another way to run big data workloads in Kubernetes. github.com/pachyderm/pachyderm. We spoke at KubeCon SF last year and v1.0 is coming out next month!
Disclosure: I work at Google on Kubernetes
Great to hear, congrats on reaching 1.0! Please do reach out when you get there, we'd love to show off your work to the community.
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#14I really wish there were more tutorials like this on how to set up Spark and other Big Data tools (TensorFlow) on cloud computing, as that personally has been the primary barrier to starting work with extra large amounts of data. (Most current tutorials require running a ton of console commands that are obsolete.) I took a Spark course on eDX last year, but the environment was set up using a customized Vagrant config…
If you're interested in Hortonworks Data Platform then check out Cloudbreak. It deploys HDP to major cloud platforms using Docker, and everything is fully open source.
http://hortonworks.com/hadoop/cloudbreak/
Disclosure: I work for Hortonworks
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#15Semi-related but does anyone know if zeppelin supports java yet?
[deleted]
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#16How would one go about this using hdfs instead of relying on gs, s3 etc. for storage? Would hdfs run as a separate k8s service?
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#17I really wish there were more tutorials like this on how to set up Spark and other Big Data tools (TensorFlow) on cloud computing, as that personally has been the primary barrier to starting work with extra large amounts of data. (Most current tutorials require running a ton of console commands that are obsolete.) I took a Spark course on eDX last year, but the environment was set up using a customized Vagrant config…
Using EMR on AWS makes it as easy as a few clicks in the AWS console to spin up a cluster running Spark and Zeppelin. Plus you get direct access to any data stored in S3 on that same AWS account..
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#18I really wish there were more tutorials like this on how to set up Spark and other Big Data tools (TensorFlow) on cloud computing, as that personally has been the primary barrier to starting work with extra large amounts of data. (Most current tutorials require running a ton of console commands that are obsolete.) I took a Spark course on eDX last year, but the environment was set up using a customized Vagrant config…
What's wrong with say Dataproc (https://cloud.google.com/dataproc/) instead of roll your own Spark? Or our (just announced) Machine Learning service based on TensorFlow (https://cloud.google.com/ml/)?
I'm all for roll your own if you're building one of these services or have existing infrastructure, but I personally like the simplicity of "here, you set this up".
Disclosure: I work at Google on Compute Engine (which underlies all of these).
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#19I really wish there were more tutorials like this on how to set up Spark and other Big Data tools (TensorFlow) on cloud computing, as that personally has been the primary barrier to starting work with extra large amounts of data. (Most current tutorials require running a ton of console commands that are obsolete.) I took a Spark course on eDX last year, but the environment was set up using a customized Vagrant config…
[deleted]
Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2
#20How would one go about this using hdfs instead of relying on gs, s3 etc. for storage? Would hdfs run as a separate k8s service?
Given the failure mode of Datanodes in Hadoop (lose X replicas, adios) you'd probably want something like the upcoming PetSet in Kubernetes 1.3. The ability to lose nodes and have that take out your job is why we push so hard on having people use our HDFS connector for GCS: it's really nice to have a high bandwidth shared object store that won't "crash".