Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Zeppelin – Interactive analytics on Spark

zeppelin.incubator.apache.org

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Apache Zeppelin – Interactive analytics on Spark

#21

Post by takluyver »

mbrzusto wrote: post #19

Notebooks where all the rage in the 90's. I spent quite a few years (~15) with Mathematica and its "notebook". Of course, Maple, Mathcad and Matlab all had the same thing. At some point, I wanted to write more readable and modular code that others could use (and that I could reuse) so I switched to using and IDE for python, Java and C. My workflow enables exploratory data analysis as well as algorithm development. Th…

> ultimately if your goal is to create a software product, learning good coding skills in an IDE is a much better path

IPython developer here. A lot of the use cases for notebooks are where you're not creating a software product. It's useful where the product you're really after is a scientific result, some plots, or the like, along with a description of how you got that. And where the product is a presentation, a demo, or documentation.

Of course, as you're doing that kind of thing, code that starts out in a notebook often becomes something you want to reuse, and you therefore move it out to an importable module. We are interested in ways to make that process more fluid.

Re: Apache Zeppelin – Interactive analytics on Spark

#22

Post by jjtheblunt »

leemoonsoo wrote: post #17
rkrzr wrote: post #2

Interesting. I wonder how this compares to an IPython notebook? While I haven't tried it, I imagine that you can use pyspark from an IPython notebook and have much of the same functionality as with Apache Zeppelin. Makes me wonder if it wouldn't have been better to just extend IPython notebooks instead of starting a new project. Edit: How to run PySpark from an IPython notebook[1] [1] http://ramhiser.com/2015/02/01/c…

I'm one of committers of Apache Zeppelin (incubating). Zeppelin is inspired by iPython notebook and many other amazing softwares that has notebook interface. I know iPython notebook has long history and large community, i really like it. Zeppelin is young, new project compare to iPython notebook. Zeppelin and iPython notebook, they are opensource. iPython notebook is lead by IPython Development Team. Zeppelin is unde…

No one seems to mention that they're all imitating Mathematica, which is still going strong since 1988 with an amazing notebook facility.

Re: Apache Zeppelin – Interactive analytics on Spark

#23

Post by sbenario »

leemoonsoo wrote: post #17
rkrzr wrote: post #2

Interesting. I wonder how this compares to an IPython notebook? While I haven't tried it, I imagine that you can use pyspark from an IPython notebook and have much of the same functionality as with Apache Zeppelin. Makes me wonder if it wouldn't have been better to just extend IPython notebooks instead of starting a new project. Edit: How to run PySpark from an IPython notebook[1] [1] http://ramhiser.com/2015/02/01/c…

I'm one of committers of Apache Zeppelin (incubating). Zeppelin is inspired by iPython notebook and many other amazing softwares that has notebook interface. I know iPython notebook has long history and large community, i really like it. Zeppelin is young, new project compare to iPython notebook. Zeppelin and iPython notebook, they are opensource. iPython notebook is lead by IPython Development Team. Zeppelin is unde…

I find it concerning when project developers don't seem to know exactly WHY the product exists or how it compares to competition on the market.

If I'm spending a lot of time working on a product that has a clear competitor(s), I'd expect to know exactly why my product is going to be better than those competitors (ie. in which ways), and what their weaknesses are.

Re: Apache Zeppelin – Interactive analytics on Spark

#24

Post by tperrigo »

I've been following Zeppelin for a while and have been actively using it for a couple of weeks (using Scala and SparkSQL/Dataframes), and although there are a few rough edges, it has been a godsend for data exploration, analysis, and feature extraction. If you're working with Spark, I highly recommend giving it a try.

Re: Apache Zeppelin – Interactive analytics on Spark

#25

Post by wmf »

sbenario wrote: post #23
leemoonsoo wrote: post #17

Earlier quoted context omitted.

I'm one of committers of Apache Zeppelin (incubating). Zeppelin is inspired by iPython notebook and many other amazing softwares that has notebook interface. I know iPython notebook has long history and large community, i really like it. Zeppelin is young, new project compare to iPython notebook. Zeppelin and iPython notebook, they are opensource. iPython notebook is lead by IPython Development Team. Zeppelin is unde…

I find it concerning when project developers don't seem to know exactly WHY the product exists or how it compares to competition on the market. If I'm spending a lot of time working on a product that has a clear competitor(s), I'd expect to know exactly why my product is going to be better than those competitors (ie. in which ways), and what their weaknesses are.

A lot of open source developers go to great lengths to avoid admitting that anything competes with anything else. I don't know if it's just conflict avoidance, rationalizing failure, or what.

Re: Apache Zeppelin – Interactive analytics on Spark

#26

Post by jamesblonde »

__database__ wrote: post #8

looks alot like the databricks cloud notebook.. looking foward to taking for a spin

Let's just say that this was out before Databricks Cloud, and it was open-source and elements of it were very similar to the demo shown at Spark summit in July 2014.

Re: Apache Zeppelin – Interactive analytics on Spark

#27

Post by codemac »

sshb wrote: post #3
rkrzr wrote: post #2

Interesting. I wonder how this compares to an IPython notebook? While I haven't tried it, I imagine that you can use pyspark from an IPython notebook and have much of the same functionality as with Apache Zeppelin. Makes me wonder if it wouldn't have been better to just extend IPython notebooks instead of starting a new project. Edit: How to run PySpark from an IPython notebook[1] [1] http://ramhiser.com/2015/02/01/c…

Also IPython is moving towards splitting the notebook part(Jupyter) and python part(IPython), so you can run any language in notebook (like IHaskell). Details are in IPython changelog: > 3.x will be the last monolithic release of IPython, as the next release cycle will see the growing project split into its Python-specific and language-agnostic components. Language-agnostic projects (notebook, qtconsole, etc.) will m…

I'm so excited for this. The power of org-babel is the common 'org table' format that can be used to interchange data between different languages.

Has IPython thought about how to send data / state between steps without python?

Re: Apache Zeppelin – Interactive analytics on Spark

#28

Post by leemoonsoo »

wmf wrote: post #25
sbenario wrote: post #23

Earlier quoted context omitted.

I find it concerning when project developers don't seem to know exactly WHY the product exists or how it compares to competition on the market. If I'm spending a lot of time working on a product that has a clear competitor(s), I'd expect to know exactly why my product is going to be better than those competitors (ie. in which ways), and what their weaknesses are.

A lot of open source developers go to great lengths to avoid admitting that anything competes with anything else. I don't know if it's just conflict avoidance, rationalizing failure, or what.

I can spend hours and hours to explain why Zeppelin is good and what is different feature compare to the other.

for example, Zeppelin has pluggable architecture so it does support not only scala but also python with built in Spark integration. Zeppelin not only support data exchange between scala and python environment but also SparkContext sharing for spark cluster resource utilization. It's got ability create rich interactive analytics GUI inside of notebook. It's got customizable layout system, and so on..

But I wanted to address more fundamental difference. Most competitors are opensource project. And as a opensource project, how community works and project direction are the most important things. That'll make huge difference in the end. And i think i explained and compared them.

Re: Apache Zeppelin – Interactive analytics on Spark

#29

Post by viklas »

leemoonsoo wrote: post #20

For someone who is interested, Here's some videos of Apache Zeppelin (Incubating) https://www.youtube.com/watch?v=QdjZyOkcG_w https://www.youtube.com/watch?v=_PQbVH_aO5E&feature=youtu.be https://www.youtube.com/watch?v=cWuWUvWVLx4 https://www.youtube.com/watch?v=xU5TBS_MsAs

The addition of angularjs is going to power some really nice dashboards. Thanks for the feature and the detailed demos (and comments on github). Looking forward to playing with this a lot!

Re: Apache Zeppelin – Interactive analytics on Spark

#30

Post by pgroth »

takluyver wrote: post #21
mbrzusto wrote: post #19

Notebooks where all the rage in the 90's. I spent quite a few years (~15) with Mathematica and its "notebook". Of course, Maple, Mathcad and Matlab all had the same thing. At some point, I wanted to write more readable and modular code that others could use (and that I could reuse) so I switched to using and IDE for python, Java and C. My workflow enables exploratory data analysis as well as algorithm development. Th…

> ultimately if your goal is to create a software product, learning good coding skills in an IDE is a much better path IPython developer here. A lot of the use cases for notebooks are where you're not creating a software product. It's useful where the product you're really after is a scientific result, some plots, or the like, along with a description of how you got that. And where the product is a presentation, a de…

This is one thing I struggle with. I usually start in a notebook and then eventually want/need to port to a module environment. It would be nice if there were some guidelines or support for this sort of transition. Any pointers?
Post reply on HN