Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

171–180 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#171

Post by jhokanson »

MrPowers wrote: post #48

Notebooks are great for invoking existing functions and exploring data. Notebooks aren't ideal for creating functions (standard text editor features are lacking and testing is impossible). Notebooks encourage an "order dependent variable assignment" programming style without abstractions. Here's what you'll commonly see in a notebook: val df = spark.read.csv("some_data") df2 = df.withColumn("clean_name", trim("name")…

The function bit has always confused me. I tend to write code with lots of functions/modules/classes for handling various aspects of the analysis and I just don't understand how that's supposed to be integrated. Instead notebooks seem better designed to handle small code snippets that rely on well known libraries. I'd be happy to jump on the notebook bandwagon but I'm having trouble seeing how I could adapt my code to the notebook style.

Re: Why Jupyter is data scientists’ computational notebook of choice

#172

Post by ChuckMcM »

I see Jupyter notebooks as the next step in spreadsheets with more code foundation and different media support. For me the interface doesn't work as well as I would like but I can see the potential.

My brother in law wants something like this for structural analysis reports where the code, data and report are all one thing that can be pulled out and examined.

Watching the new iPad announcement today I think this is something that would make an excellent iPad app as well.

Re: Why Jupyter is data scientists’ computational notebook of choice

#173

Post by TeMPOraL »

benjaminjackman wrote: post #120
amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

I think they fundamentally json is just the wrong format for these files. Speaking from (ancient and limited) experience I made a little notebook-style interpreter for learning scala back in 2009 or so called scalide. It saved its files ("scalapads") to XML. XML actually worked better in some ways since most of the code could live between the tags unescaped (sans &) so it merged / diffed the user code well. The meta-…

The closest thing I've seen to what you described would be... Emacs. It actually uses the "metadata in file-specific comments" paradigm. You can put file-local values for Emacs variables in comments at the top or bottom of your file, like described in [0].

Your example could be rewritten as:

  # -*- notebook-lang: python -*-
or

  // -*- notebook-lang: scala -*-
Still, the usual way of using Emacs for "interactive notebooks" is via org-mode, which is a better Markdown with support for (among other things) executing code blocks straight in the org document you're writing. This way, Emacs support all your points 1 to 5, and is generally more powerful than Jupyter or other similar things, but it also means you can kiss any kind of collaboration goodbye.

For some weird reason, the more powerful a tool, the less likely it is other people will be using it.

--

[0] - https://www.gnu.org/software/emacs/manual/html_node/emacs/Sp...

Re: Why Jupyter is data scientists’ computational notebook of choice

#174

Post by solomatov »

kprybol wrote: post #146
solomatov wrote: post #118

Earlier quoted context omitted.

If you are interested in workbooks which are collaborative and versioned, take a look at http://datalore.io/ Version control is transparent and integrated and it's possible to work with workbooks collaboratively.

Any chance of this being offered for on-prem install in the future? Looks interesting but cloud only makes it a no go for my team.

We are seriously considering such a possibility. Do you have any specific requirements for on prem installation?

Re: Why Jupyter is data scientists’ computational notebook of choice

#175

Post by solomatov »

ChuckMcM wrote: post #172

I see Jupyter notebooks as the next step in spreadsheets with more code foundation and different media support. For me the interface doesn't work as well as I would like but I can see the potential. My brother in law wants something like this for structural analysis reports where the code, data and report are all one thing that can be pulled out and examined. Watching the new iPad announcement today I think this is s…

If you are interested in spreadsheets but with Python, try http://datalore.io/ It tracks dependencies automatically and recalculates them when needed (this behavior can also be made less automatic).

Re: Why Jupyter is data scientists’ computational notebook of choice

#176

Post by azag0 »

ChuckMcM wrote: post #172

I see Jupyter notebooks as the next step in spreadsheets with more code foundation and different media support. For me the interface doesn't work as well as I would like but I can see the potential. My brother in law wants something like this for structural analysis reports where the code, data and report are all one thing that can be pulled out and examined. Watching the new iPad announcement today I think this is s…

Already now you can run a Jupyter notebook on an http server and serve it to Safari in an iPad.

Re: Why Jupyter is data scientists’ computational notebook of choice

#177

Post by selimthegrim »

ChuckMcM wrote: post #172

I see Jupyter notebooks as the next step in spreadsheets with more code foundation and different media support. For me the interface doesn't work as well as I would like but I can see the potential. My brother in law wants something like this for structural analysis reports where the code, data and report are all one thing that can be pulled out and examined. Watching the new iPad announcement today I think this is s…

Your brother in law’s wish is granted. Blockpad is specifically targeted towards such reports

https://blockpad.net

Re: Why Jupyter is data scientists’ computational notebook of choice

#178

Post by jimhefferon »

Desustorm wrote: post #139
amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

I have used jupytext ( https://github.com/mwouts/jupytext ) for this and it seems to work great - it outputs a separate .py file which is easily diff-able.

Thanks for the note; this looks good.

Re: Why Jupyter is data scientists’ computational notebook of choice

#179

Post by rb808 »

ChuckMcM wrote: post #172

I see Jupyter notebooks as the next step in spreadsheets with more code foundation and different media support. For me the interface doesn't work as well as I would like but I can see the potential. My brother in law wants something like this for structural analysis reports where the code, data and report are all one thing that can be pulled out and examined. Watching the new iPad announcement today I think this is s…

I used to use Excel extensively. I've started using Jupyter as a replacement. Some things are great like python modules that can do anything, and visualizations. But if there are less than 100k rows, its still much easier to just use Excel. I'm kinda disappointed. IF you have more than 100k rows then Excel starts to be cumbersome. That is the sweet spot for me.

Re: Why Jupyter is data scientists’ computational notebook of choice

#180

Post by TeMPOraL »

tmalsburg2 wrote: post #155
tony_cannistra wrote: post #21

Earlier quoted context omitted.

there’s a relatively esoteric paradigm known as “literate programming” which has been around since Knuth (he wrote the book [0]) and that has some software tools associated, of which Jupyter is a particularly web-age example. [0]: https://en.m.wikipedia.org/wiki/Literate_programming

Literate programming was esoteric, true, but the concept saw a huge renaissance in academia and data science with the advent of RMarkdown¹ which for many of my colleagues is the default way of preparing technical documents. Another area in which literate programming has become hugely popular is Emacs' Org-mode ecosystem which has fantastic support in the form of Org Babel². I use literate programming for almost every…

I'm still not sold on writing complete program this way (I did try, with various level of success), but even partially-literate approach is ridiculously convenient if you happen to live in Emacs.

I do my task management and note-taking in Org-mode, and recently I found myself doing things like jotting in the middle of my notes[0]:

  #+BEGIN_SRC http
    GET address.to.api:123/sth
  #+END_SRC
and tapping CTRL+C twice, to get the actual response of the API I was debugging.

Or, the other day I was making notes about gravity batteries, and was wondering how efficient is one startup's solution. I briefly thought about firing up Jupyter, but then simply wrote the following[1]:

   these guys power a LED (or three?) with a 0.1W, generated through dropping
   a 12kg weight down 1.8 meters over 20 minutes.

   Doing some basic math on that:
   #+BEGIN_SRC elisp
     (let* ((m 12)
            (g 9.81)
            (h 1.8)
            (_t (* 20 60))
            (E (* m g h))                    ; E = m*g*h
            (P (/ E _t))                     ; P = E/t
            (efficiency (/ 0.1 P))           ; efficiency = Pout/Pin
            )
       `("ideal power [W]" ,P
         "efficiency [1]" ,efficiency))
   #+END_SRC
Typing CTRL+C twice, out pops:

  #+RESULTS:
  | ideal power [W] | 0.17658000000000001 | efficiency [1] | 0.5663155510250312 |
(which is automatically rendered as an org-mode table I can operate on, or even reference in other code snippets).

Point being, note-taking in org mode makes it ridiculously easy to invoke any programming language you hooked up to Emacs without breaking your flow, and you get to edit the code in the mode specific to that programming language - so everything from autocomplete to linters work.

I know Emacs is niche, but I can't recommend it enough.

--

[0] - BEGIN/END_SRC block is under convenient autocomplete of "[1] - this is a real note, so if I got the physics wrong, I just made a fool of myself publicly -.-

Post reply on HN