Why Jupyter is data scientists’ computational notebook of choice • Hacker News Board

Re: Why Jupyter is data scientists’ computational notebook of choice

#301

Post by cbkeller » Thu, Nov 01, 2018, 4:20 AM UTC

Insanity wrote: post #271

cbkeller wrote: post #199

Earlier quoted context omitted.

> many academic scientists have not been socialized to make a distinction between development and production environments Geologist here - definitely true in my field. Nonetheless, while I don't develop in notebooks at all, I do use them for "reproducibility" in a sense -- by putting a bit of dependency info in a github repo along with a .ipynb file, I can do things like this: https://mybinder.org/v2/gh/brenhinkeller…

Don't know if it is something you actively need, but the image did not load for me. (Even after re-evaluating the cell). The image after: "For example (KJ04-70)" (I also re-ran the preceding cells).

Thanks for the tip! Interestingly enough, this appears to be browser-dependent. Apparently including a pdf in notebook markdown using an img tag like

works in Safari but not Chrome or Firefox. Switched to SVGs for now.

So much for reproducibility :/

Re: Why Jupyter is data scientists’ computational notebook of choice

#302

Post by lispm » Thu, Nov 01, 2018, 9:41 PM UTC

throwaway487548 wrote: post #250

Jupyter could be viewed as a modern reminiscence of Lisp Machine UI, without the elegance of homoiconicity, of course. Python is a good "glue" for optimized C++ or fortran libraries which the core of things like tensorflow or numpy. Everything fits together nicely.

> Jupyter could be viewed as a modern reminiscence of Lisp Machine UI

maybe some part of it, but the Lisp Machine UI has a full window system, many different applications based on it with different UIs (font editor, file system browser, process overview, chat program, terminal, Zmacs editor, debugger, documentation browser, documentation editor, drawing program, ...)

Re: Why Jupyter is data scientists’ computational notebook of choice

#303

Post by dual_basis » Fri, Nov 02, 2018, 12:20 PM UTC

lispm wrote: post #302

throwaway487548 wrote: post #250

Jupyter could be viewed as a modern reminiscence of Lisp Machine UI, without the elegance of homoiconicity, of course. Python is a good "glue" for optimized C++ or fortran libraries which the core of things like tensorflow or numpy. Everything fits together nicely.

> Jupyter could be viewed as a modern reminiscence of Lisp Machine UI maybe some part of it, but the Lisp Machine UI has a full window system, many different applications based on it with different UIs (font editor, file system browser, process overview, chat program, terminal, Zmacs editor, debugger, documentation browser, documentation editor, drawing program, ...)

That is what Jupyter Lab is becoming.

Re: Why Jupyter is data scientists’ computational notebook of choice

#304

Post by haskellandchill » Fri, Nov 02, 2018, 7:21 PM UTC

peatmoss wrote: post #159

haskellandchill wrote: post #123

Earlier quoted context omitted.

Woa, thanks for pointing out Zelig, I needed that relogit and I didn't even know it :)

The counter factual simulation features are amazing and easy.

Wondering why it hasn't got more publicity?

Re: Why Jupyter is data scientists’ computational notebook of choice

#305

Post by lispm » Sat, Nov 03, 2018, 10:36 AM UTC

dual_basis wrote: post #303

lispm wrote: post #302

Earlier quoted context omitted.

> Jupyter could be viewed as a modern reminiscence of Lisp Machine UI maybe some part of it, but the Lisp Machine UI has a full window system, many different applications based on it with different UIs (font editor, file system browser, process overview, chat program, terminal, Zmacs editor, debugger, documentation browser, documentation editor, drawing program, ...)

That is what Jupyter Lab is becoming.

great, but how is it Lisp Machine UI like?

Re: Why Jupyter is data scientists’ computational notebook of choice

#306

Post by nemoniac » Sat, Nov 03, 2018, 1:42 PM UTC

TeMPOraL wrote: post #173

benjaminjackman wrote: post #120

Earlier quoted context omitted.

I think they fundamentally json is just the wrong format for these files. Speaking from (ancient and limited) experience I made a little notebook-style interpreter for learning scala back in 2009 or so called scalide. It saved its files ("scalapads") to XML. XML actually worked better in some ways since most of the code could live between the tags unescaped (sans &) so it merged / diffed the user code well. The meta-…

The closest thing I've seen to what you described would be... Emacs. It actually uses the "metadata in file-specific comments" paradigm. You can put file-local values for Emacs variables in comments at the top or bottom of your file, like described in [0]. Your example could be rewritten as: # -*- notebook-lang: python -*- or // -*- notebook-lang: scala -*- Still, the usual way of using Emacs for "interactive noteboo…

Well, the reason is not so weird. Generally speaking, the more powerful the tool, the higher the bar. More time and effort is required to learn it and become proficient with it. When it comes to the very powerful tools, few will have the aptitude or be prepared to put in the effort to learn them.

For those who don't, less powerful tools take their place and proliferate.

As you say, emacs checks all the boxes but the majority is not prepared to learn it and prefers to program throught their browser.

Re: Why Jupyter is data scientists’ computational notebook of choice

#307

Post by jxramos » Mon, Nov 05, 2018, 8:52 PM UTC

"Two additional tools have enhanced Jupyter’s usability. One is JupyterHub, a service that allows institutions to provide Jupyter notebooks to large pools of users. The IT team at the University of California, Berkeley, where Pérez is a faculty member, has deployed one such hub, which Pérez uses to ensure that all students on his data-science course have identical computing environments. “We cannot possibly manage IT support for 800 students, helping them debug why the installation on their laptop is not working; that’s simply infeasible,” he says." I think this result is a real winner, I recall the problems of setup in university student labs. Good win for reducing teaching friction.

Re: Why Jupyter is data scientists’ computational notebook of choice

#308

Post by ISL » Thu, Nov 08, 2018, 4:46 PM UTC

kristjansson wrote: post #238

ISL wrote: post #170

Earlier quoted context omitted.

I'm not a Jupyter user, but I solve the reproducibility problem with Make. As a project moves from exploration toward production, the entire thing is wrapped into a Makefile that can flow from raw data to publication in a single call to make.

This has been my solution as well. There’s little that feels as good as running `make -B report` and watching the whole thing be rebuilt from scratch. How do you manage encapsulating each step, and passing data between them?

I don't do it well. See

https://github.com/4kbt/ReplicableAnalysis

and

https://github.com/4kbt/PlateWash

as examples. The former is smaller/less complicated. The latter was my thesis work -- more complicated and (unfortunately) abuses recursive calls to Make.