Why Jupyter is data scientists’ computational notebook of choice • Hacker News Board

Re: Why Jupyter is data scientists’ computational notebook of choice

#81

Post by taeric » Tue, Oct 30, 2018, 3:24 PM UTC

makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

You parting "Granted..." is precisely what fills me with dread when I see notebooks. Yes, I have seen poorly done source files. I made more than a few myself. However, many of the practices we have grown into as sound programming advice seem to be largely thrown out the window for these notebooks.

The irony, to me, is that I actually typically argue for the mixing of presentation and content. But to me, notebooks look like an attempt by people to make a WYSIWYG out of JUnit/TestNG/whatever style reports. Only, without the repeatability.

There is also the entire bend where these are taking off in a way that doesn't make sense. Do they do the things you are saying? Well, yeah. But no better than plenty of tools before them. Mathematica and Matlab both had "notebook" like features for a long long time. Complete with optimized libraries. And this is ignoring the interactivity of the old LISP machines. (You can see from my history I have a soft spot for emacs org-mode.)

Jupyter is a lot of things. Bad isn't necessarily one of them, but exceptional isn't, either. Heavily marketed is.

Re: Why Jupyter is data scientists’ computational notebook of choice

#82

Post by klmr » Tue, Oct 30, 2018, 3:25 PM UTC

makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

I fundamentally agree with you but I have the feeling that some some of the major proponents of notebooks belong to the category of people who misunderstand them, and simply use them for everything, and write long and convoluted notebooks; I’ve definitely seen my share of those in my domain (bioinformatics, AI) and elsewhere. By contrast, Joel Grus for instance perfectly understands their strengths and weaknesses.

As for being a a good REPL, I feel that an actual REPL (+ editor integration) works better than notebooks: You can combine a literate document with a REPL but still get the benefits of a proper editor/IDE and a proper execution environment, rather than a half-hearted mix of both that’s hosted inside a HTML contenteditable (= Jupyter), and you also get “charts, interactivity, nice and wide sortable tables, etc” if you want). RMarkdown inside RStudio or Nvim-R does this well. — I just don’t want to give up the advantages of a proper editor for the very slight increase in integration that Jupyter gives me.

Re: Why Jupyter is data scientists’ computational notebook of choice

#83

Post by i-am-charmander » Tue, Oct 30, 2018, 3:30 PM UTC

dev_dull wrote: post #79

amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

"Production notebooks" should not be a thing... Unless prefaced by a bold RUN ALL.

Re: Why Jupyter is data scientists’ computational notebook of choice

#84

Post by dimatura » Tue, Oct 30, 2018, 3:30 PM UTC

trop wrote: post #66

Question/idea: Could a notebook-model supplant bespoke photographing processing software such as the "darkroom" mode of Lightroom (or darktable). The extant programs essentially take a lot of data (camera's raw output) and apply a configurable recipe to produce intelligible output (an image). Each recipe (stored as an XMP sidecar) is essentially a list of math operations (increase brightness, wavelet decompose, chang…

Yes and no. I do computer vision and like photography as well. I use Jupyter notebooks extensively for computer vision, and they work pretty well for (semi) interactive manipulation of image data with code. But as a general purpose tool, it's too clunky for anything more than prototyping. I don't see them replacing darktable/lightroom anytime soon.

Re: Why Jupyter is data scientists’ computational notebook of choice

#85

Post by taeric » Tue, Oct 30, 2018, 3:31 PM UTC

glup wrote: post #8

It's easy to grade student assignments in notebooks with https://github.com/jupyter/nbgrader , which also makes it great for teaching.

Meh. As long as you have defined deliverables between grader and student, grading programming based assignments are relatively easy. Coursera has been around longer than Jupyter has been popular, after all. (And they aren't all just multiple choice.)

Being interactive is what makes it good for teaching. But there are plenty of interactive options. And for a certain class of teaching, it is not "on rails" enough such that people will have to have a ramp up period first on Jupyter before they can really get into their topic.

Re: Why Jupyter is data scientists’ computational notebook of choice

#86

Post by peatmoss » Tue, Oct 30, 2018, 3:31 PM UTC

mattkrause wrote: post #43

wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

It seems to depend on what you’re doing. Python definitely has more mindshare for machine learning, and particularly deep learning. However, that’s not all of statistics. For things like mixed-effects modeling, I think R still has a clear lead. There are some python packages (e.g., statsmodels) but R’s lme4 has more features, like custom covariance structures, and virtually every textbook and tutorial currently uses…

And then there is the Zelig modeling framework for R that I can’t imagine not using after having used it.

Don’t get me wrong, I like Python well enough, and knew it before I coded R. But Python is really behind R in stats support. I’d also add the tidyverse in there for general data munging.

If I want libraries I’ll use R; if I want a programming language I love I'll use Racket or maybe Clojure; if I want some libraries and an okay programming language I’ll use Python, I guess.

Re: Why Jupyter is data scientists’ computational notebook of choice

#87

Post by yodon » Tue, Oct 30, 2018, 3:32 PM UTC

pavanagrawal123 wrote: post #57

projectramo wrote: post #35

Here are the issues with Jupyter, and most other flavor, of notebook: 1. variables have to be explicitly output The most important tool for programming, for me, is that window that shows you the current state of all the variables. When I step through a program, I look at the state. 90% of my debugging solutions come from seeing that variable doesn't have the right state. 2. Intellisense For the love of god, I do not…

Hey There! I'm trying to solve the issue of IntelliSense.. I'm building/improving Jupyter Notebooks inside VSCode: https://github.com/pavanagrawal123/VSNotebooks . It's a fork from another extension somebody already built, but all activity is dead, so I'm starting up dev on an active fork. I'd love to hear any feedback y'all have! :) Also planning to add some nice debug features, plus hopefully integration into the i…

How do you see the idea of a vs code notebook comparing to or being different from the goals of the hydrogen editor?

Re: Why Jupyter is data scientists’ computational notebook of choice

#88

Post by fjuerfilis » Tue, Oct 30, 2018, 3:33 PM UTC

currymj wrote: post #58

saosebastiao wrote: post #44

Earlier quoted context omitted.

I'm definitely well in the R camp but keep feeling this nagging pull from Python. Especially for trading...it would be so nice to have a language for both research and production, as right now I translate all my research into scala for production.

I hate to be the stereotypical Julia recommender, but it is made for this use case, more so than Python, which isn’t all that much faster than R if speed matters. (Unless you want to try Cython but that’s a whole bag of worms.)

I'd second that. R and Python both have the same pre-LLVM performance issues.

I don't expect either R or Python to go away either time soon, nor would I want them to, but I would like to see people moving to things like Julia and Nim, which have the same level of expressivity, but are much more performant. I have difficulty imagining many people saying "I love programming in R and Python, but don't like Julia or Nim."

I like Python but at least with stats/numerics there isn't a big reason to move away from R except for specific libraries (especially DL stuff) or front-end integration with web-land (and even then things like Jupyter mitigate against that).

Re: Why Jupyter is data scientists’ computational notebook of choice

#89

Post by bitL » Tue, Oct 30, 2018, 3:34 PM UTC

dev_dull wrote: post #79

amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

You don't use Jupyter notebooks in production; they are super useful for pitching ideas to clients/bosses and doing some early prototyping. I feel sorry for anyone that has to work with "pure data scientists" that have no clue about software engineering practices...

Re: Why Jupyter is data scientists’ computational notebook of choice

#90

Post by jrumbut » Tue, Oct 30, 2018, 3:38 PM UTC

makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

I think it speaks to people's desire for a quick and easy to set up basic GUI creator with an editor that allows inline code editing, and no need to deal explicitly with the client server interaction.

I myself, as someone who likes to create really solid and maintainable tools, have fallen into the notebook trap and written things like "change the month in cell 22 then execute cells 1 through 3 and 20 through 27 to update the report".

The notebook format was great for prototyping what was really a small app. You don't really have those problems when you're just generating a document.