Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

131–140 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#131

Post by natalyarostova »

entee wrote: post #122
dev_dull wrote: post #79

Earlier quoted context omitted.

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

At our company if it's in a notebook it's not considered ready for production, it must run as a script before being considered for Eng to take over from DS. It's actually not that hard to write a notebook in such a way that it converts easily to a script. Just check and make sure that your variables/functions/whatever are initialized above the cell(s) they're used in, declare all imports in the top cell, and periodic…

I think cleanup is part of it. I also have noticed as a guy on the DS job family, but who has taken a large interest in SDE work, separating the job families can result in churn. For example, I might think of three model choices A, B and C. C may be the worst of the three, but only very marginally worse. It can also be the case that C is an order of magnitude easier to keep and maintain in production.

I've seen cases where the wrong choice here ends up requiring three SDEs for half a year, where if they gave up a tiny benefit of the best model, they could have done it with 1 SDE in 1 month.

Re: Why Jupyter is data scientists’ computational notebook of choice

#132

Post by celrod »

jonnycomputer wrote: post #93
makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

If you ever have used an R Notebook written in R-Markdown, then its pretty easy to see why Jupyter Notebooks putting everything in JSON is just... infuriatingly wrong-headed. In an R Notebook, I can see my code, I can see my text, everything is exceedingly simple to understand, and I can edit it in any of the fantastic text editors out there (Jupyter's editor is not among them)

RStudio is also my favorite editor. All my work is data science / stats related, where I like the workflow of writing/modifying code in a .R (or .py) file, and being able to quickly experiment by running chunks in a REPL with Ctrl + Enter.

R and Python are supported. No Julia, unfortunately. VS Code and Atom support similar workflows with Julia. However, the Julia Language server in VS Code is extremely unstable and I regularly lose LaTeX completions. The REPL in Atom is mind boggling laggy and slow to the point that it is much less frustrating to copy and paste code into a REPL running in your favorite terminal emulator.

Re: Why Jupyter is data scientists’ computational notebook of choice

#133

Post by SiempreViernes »

reilly3000 wrote: post #127

Does anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soo…

Do you have some server to host it on?

If so, I’d run the notebook on the remote server and just teach them whatever command they need to make a ssh tunnel there. Something like what is described here: https://techtalktone.wordpress.com/2017/03/28/running-jupyte...

So they would utter the unknowable incantation and then point their browser at localhost:8000 or whatever and then use their version of the notebook.

Re: Why Jupyter is data scientists’ computational notebook of choice

#134

Post by mimischi »

short_sells_poo wrote: post #64
daveFNbuck wrote: post #59

Earlier quoted context omitted.

When you're processing a lot of data, it can be expensive to keep re-running your whole script every time you make a change. The notebook keeps the results of your earlier steps in memory when you want to change and re-run a later step. This is a trade-off between how much code you're writing and how much data you're processing. If you're writing maybe 20 lines of code but you have enough input that it takes several…

So does the standard terminal repl in python. You can achieve the same workflow with having a plain old python file, and then just use your favorite editor's "Send block of code to console" function. This way, you retain your editor's functionality while you can work just as interactively as with a notebook.

But then there are plotting and interactive widgets in the notebook.

Re: Why Jupyter is data scientists’ computational notebook of choice

#135

Post by mike_ivanov »

thousandautumns wrote: post #76
wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

I'm not sure what about Jupyter makes Python more compelling in comparison to R. R is entirely usable in Jupyter Notebooks, and R Notebooks are, in my opinion, possibly superior to Jupyter notebooks in many ways. > and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions I'm not sure what you are…

s/possibly/definitely/

Re: Why Jupyter is data scientists’ computational notebook of choice

#136

Post by Immortalin »

jrumbut wrote: post #90
makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

I think it speaks to people's desire for a quick and easy to set up basic GUI creator with an editor that allows inline code editing, and no need to deal explicitly with the client server interaction. I myself, as someone who likes to create really solid and maintainable tools, have fallen into the notebook trap and written things like "change the month in cell 22 then execute cells 1 through 3 and 20 through 27 to u…

Yes. It's called Microsoft Excel. Software engineers don't like VB for the same reason they don't like Python-in-a-notebook but you cannot deny its effectiveness.

Re: Why Jupyter is data scientists’ computational notebook of choice

#137

Post by hogu »

reilly3000 wrote: post #127

Does anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soo…

I'm literally building this right now to fulfill this need. If you shoot me an email to hugo@opensourceanswers.com, I can let you know when it's ready. My plan is to charge a premium (similar to github prices) per user, and pass on compute costs directly to the customer with no markup

Re: Why Jupyter is data scientists’ computational notebook of choice

#138

Post by jboggan »

dev_dull wrote: post #79
amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

I'm coming to realize one of the key skills for a data engineer to have nowadays is "productionizing" notebook code from data scientists and PMs and teaching them to make it more testable and modular in the first place.

Re: Why Jupyter is data scientists’ computational notebook of choice

#139

Post by Desustorm »

amirathi wrote: post #65

Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/

I have used jupytext (https://github.com/mwouts/jupytext) for this and it seems to work great - it outputs a separate .py file which is easily diff-able.
Post reply on HN