Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2 • Hacker News Board

Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2

#21

Post by BrandonBradley » Thu, Mar 31, 2016, 4:33 AM UTC

minimaxir wrote: post #3

I really wish there were more tutorials like this on how to set up Spark and other Big Data tools (TensorFlow) on cloud computing, as that personally has been the primary barrier to starting work with extra large amounts of data. (Most current tutorials require running a ton of console commands that are obsolete.) I took a Spark course on eDX last year, but the environment was set up using a customized Vagrant config…

I've found Spark on Mesos 'fairly easy' to setup. Fairly easy after quite of bit of pain and configuration management.

Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2

#22

Post by ptrincr » Thu, Mar 31, 2016, 5:57 AM UTC

minimaxir wrote: post #5

ryanl-ee wrote: post #4

Earlier quoted context omitted.

Have you checked out DCOS? Once you have your cluster set up, your path to installing Spark is running "dcos package install spark" on your laptop.

Huh, no I haven't. Although the package support appears to be more narrow than with Kubernetes/Docker, but DCOS is a possibility too.

You should also check out Ambari, fantastic open source Hadoop/spark cluster management tool.

Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2

#23

Post by markvdb » Thu, Mar 31, 2016, 6:51 AM UTC

Totally off topic, but...

Am I the only one to think of https://en.wikipedia.org/wiki/Hindenburg_disaster when seeing the words "spark" and "zeppelin" together? (This type of air ship is called a "zeppelin" in my native language...)

Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2

#24

Post by mindcrime » Thu, Mar 31, 2016, 8:24 AM UTC

ptrincr wrote: post #22

minimaxir wrote: post #5

Earlier quoted context omitted.

Huh, no I haven't. Although the package support appears to be more narrow than with Kubernetes/Docker, but DCOS is a possibility too.

You should also check out Ambari, fantastic open source Hadoop/spark cluster management tool.

Second the motion. Ambari is actually one of the key elements of the tech stack we're using for our service. With the "Blueprints" feature and the REST API, it's a really convenient way to automate the provisioning of Spark/Hadoop clusters.

Even better, Ambari isn't actually limited to installing just Hadoop/Spark, etc. In principle, you could extend it to take care of installing pretty much anything.

Re: Using Spark and Zeppelin to Process Big Data on Kubernetes 1.2

#25

Post by ryanl-ee » Thu, Mar 31, 2016, 3:01 PM UTC

minimaxir wrote: post #5

ryanl-ee wrote: post #4

Earlier quoted context omitted.

Have you checked out DCOS? Once you have your cluster set up, your path to installing Spark is running "dcos package install spark" on your laptop.

Huh, no I haven't. Although the package support appears to be more narrow than with Kubernetes/Docker, but DCOS is a possibility too.

Well, the included packages in the "dcos package" CLI repo are just the prepackaged frameworks. You can actually deploy Kubernetes on top of DCOS in the same way - it natively supports Docker containers as an execution method.