Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

121–130 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#121

Post by cwyers »

bitL wrote: post #89
dev_dull wrote: post #79

Earlier quoted context omitted.

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

You don't use Jupyter notebooks in production; they are super useful for pitching ideas to clients/bosses and doing some early prototyping. I feel sorry for anyone that has to work with "pure data scientists" that have no clue about software engineering practices...

It depends on what you're doing, yeah? In RMarkdown notebooks... yeah, I wouldn't write models in one. But if the focus is on embedding some visualizations and tables into a document, and then refreshing the document to every so often pull in new data, I can see that as a production use for a notebook. TL;DR: Can be useful for reporting, wouldn't use it anywhere else in the pipeline.

Re: Why Jupyter is data scientists’ computational notebook of choice

#122

Post by entee »

dev_dull wrote: post #79
amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

At our company if it's in a notebook it's not considered ready for production, it must run as a script before being considered for Eng to take over from DS. It's actually not that hard to write a notebook in such a way that it converts easily to a script. Just check and make sure that your variables/functions/whatever are initialized above the cell(s) they're used in, declare all imports in the top cell, and periodically move cells to fix any inconsistencies with these rules (checking that you didn't break anything of course). I've always said that Data Scientist doesn't mean, "I don't do engineering," good basic eng practice helps make more productive data science and brings it into production more robustly. How do you know your models work well if the code that generated them is inscrutable?

I wonder how much of the "3 engineers for 1 data scientist" ratio I hear all the time is due to Data Engineering being assigned the role of cleanup to code that should be better in the first place.

Re: Why Jupyter is data scientists’ computational notebook of choice

#123

Post by haskellandchill »

peatmoss wrote: post #86
mattkrause wrote: post #43

Earlier quoted context omitted.

It seems to depend on what you’re doing. Python definitely has more mindshare for machine learning, and particularly deep learning. However, that’s not all of statistics. For things like mixed-effects modeling, I think R still has a clear lead. There are some python packages (e.g., statsmodels) but R’s lme4 has more features, like custom covariance structures, and virtually every textbook and tutorial currently uses…

And then there is the Zelig modeling framework for R that I can’t imagine not using after having used it. Don’t get me wrong, I like Python well enough, and knew it before I coded R. But Python is really behind R in stats support. I’d also add the tidyverse in there for general data munging. If I want libraries I’ll use R; if I want a programming language I love I'll use Racket or maybe Clojure; if I want some librar…

Woa, thanks for pointing out Zelig, I needed that relogit and I didn't even know it :)

Re: Why Jupyter is data scientists’ computational notebook of choice

#124

Post by brylie »

bitL wrote: post #89
dev_dull wrote: post #79

Earlier quoted context omitted.

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

You don't use Jupyter notebooks in production; they are super useful for pitching ideas to clients/bosses and doing some early prototyping. I feel sorry for anyone that has to work with "pure data scientists" that have no clue about software engineering practices...

FWIW, Netflix uses Jupyter notebooks in production, using nteract UI:

https://medium.com/netflix-techblog/notebook-innovation-591e...

https://nteract.io/

This approach seems promising, particularly as it facilitates cross-disciplinary collaboration.

Re: Why Jupyter is data scientists’ computational notebook of choice

#125

Post by williamstein »

thousandautumns wrote: post #76
wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

I'm not sure what about Jupyter makes Python more compelling in comparison to R. R is entirely usable in Jupyter Notebooks, and R Notebooks are, in my opinion, possibly superior to Jupyter notebooks in many ways. > and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions I'm not sure what you are…

> R is entirely usable in Jupyter Notebooks

Not for everybody, e.g., the Swirl R package (https://swirlstats.com/) doesn't work in Jupyter, since Jupyter has limited support for R's many ways of getting interactive input from users.

Re: Why Jupyter is data scientists’ computational notebook of choice

#126

Post by renjimen »

thealfreds wrote: post #45
renjimen wrote: post #42

Earlier quoted context omitted.

That's surprising because I have the opposite experience! Since my first cell is to import all of the libraries I want to use to memory, the intellisense works without fail, regardless of how big the libraries are. Comparing that with my VS Code experience where using intellisense to pull up functions' doc strings takes an age for all but the inbuilt Python libraries.

I'm not a Python dev. Is it not common to just type and let it auto import in the required libraries for you?

You'd think so. Maybe my setup is faulty. Something for me to look in to

Re: Why Jupyter is data scientists’ computational notebook of choice

#127

Post by reilly3000 »

Does anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soon I found myself installing helm and trying to follow along a tutorial on how to deploy JupyterHub on a kubernetes cluster. That started to add an unmanageable level of complexity to own, especially to share a simple notebook. And while spinning up a GKE node per user is the whole point of Kubernetes, it got expensive quickly in my test. We cannot spend $75K a year on Domino. Any other options?

Re: Why Jupyter is data scientists’ computational notebook of choice

#129

Post by zimablue »

amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

Jupytext linked elsewher eont he thread seems like a step in the right direction. Instead of changing the whole tool, accept that you're always going to be married to github and change the serialization-layer to be source control friendly. Basically, split the input from the output+metadata and flatten it all to text. Then you can source control it fine and if you need use the output+metadata fold them back in.

Re: Why Jupyter is data scientists’ computational notebook of choice

#130

Post by reilly3000 »

reilly3000 wrote: post #127

Does anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soo…

After laying that all out there I now realize I could have just given my handful of users their own files. Doh.

Still, the fact remains that JuptyerHub is powerful, but difficult to install and manage if you're not a university IT dept. Any SMB solutions?

Post reply on HN