Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Git and Jupyter Notebooks play nice

timstaley.co.uk

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Making Git and Jupyter Notebooks play nice

#11

Post by amirathi »

A year ago, I was frustrated (and surprised) to see one can't do code reviews with Jupyter Notebooks. GitHub diffs for notebooks JSON are super messy.

I set out to build ReviewNB[1], code review tool for Jupyter Notebooks. Turns out a lot of other people had this exact problem. One can see visual diffs & write review comments on notebook cell. Currently only works with GitHub though.

If you want to diff locally (before committing changes), you might like nbdime[2].

[1] https://www.reviewnb.com/

[2] https://github.com/jupyter/nbdime

Re: Making Git and Jupyter Notebooks play nice

#13

Post by enriquto »

kevcampb wrote: post #5

Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells

Jupytext is great. What the world needs is that it becomes--transparently--the default and we can get rid of the silly un-editable json format.

Re: Making Git and Jupyter Notebooks play nice

#14

Post by morotter »

I found a good compromise by using VSCode Python extension. You can import Jupyter notebooks as Python scripts and the other way around [0]. If I need to work on a notebook, I prefer working with a Python script and the interactive window [1]. Then I commit both script and ipynb version.

[0] https://code.visualstudio.com/docs/python/jupyter-support-py...

[1] https://code.visualstudio.com/docs/python/jupyter-support-py...

Re: Making Git and Jupyter Notebooks play nice

#15

Post by kbumsik »

Can Jupyter stop using JSON and look for a new structure (not only about the file format but also the data fields representation)? The current format is unordered and contains huge binary blobs which makes it very inefficient and version control is simply a pain.

I believe a new design worth sacrificing backward compability.

Re: Making Git and Jupyter Notebooks play nice

#16

Post by bobbylarrybobby »

morotter wrote: post #14

I found a good compromise by using VSCode Python extension. You can import Jupyter notebooks as Python scripts and the other way around [0]. If I need to work on a notebook, I prefer working with a Python script and the interactive window [1]. Then I commit both script and ipynb version. [0] https://code.visualstudio.com/docs/python/jupyter-support-py... [1] https://code.visualstudio.com/docs/python/jupyter-support-p…

Agree 100%. Only thing I’d add is that the killer feature of the extension is that it allows one to treat ordinary python files as notebooks (without converting!) by 1. Connecting to a persistent kernel instance 2. Allowing the user to use magic comments to delineate code cells within the python file. With these two things, and the ability to export as a real notebook, you get the great experience of a notebook — submitting easily editable cells to a kernel one at a time, as many times as you want — without of all the usual baggage that that would entail. Plus you get to edit in a decent editor (and edit things other than just python files in it) instead of the crap Jupyter forces you to use.

Re: Making Git and Jupyter Notebooks play nice

#17

Post by Iv »

kevcampb wrote: post #8
Iv wrote: post #7

Earlier quoted context omitted.

I guess you write hooks then, you don't convert on every commit?

Jupytext works as an extension in Jupyter. Your work is saved as py automatically whenever you work on the notebook. So it's not a conversion at commit time, unlike say nbstripout

Oooohhhh! Nice! I'll install it right away!

Re: Making Git and Jupyter Notebooks play nice

#18

Post by nabdab »

kevcampb wrote: post #5

Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells

I love jupytext, but i feel like it’s a patch on a problem that should have just been solved. Just change jupyter to work directly in the genereres file format and skip the “pair files” hassle.

Re: Making Git and Jupyter Notebooks play nice

#19

Post by leni536 »

nabdab wrote: post #18
kevcampb wrote: post #5

Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells

I love jupytext, but i feel like it’s a patch on a problem that should have just been solved. Just change jupyter to work directly in the genereres file format and skip the “pair files” hassle.

Or it could be solved at the git tooling side by introducing Jupyter specific merge and diff tools.

Re: Making Git and Jupyter Notebooks play nice

#20

Post by TimSAstro »

Hi, HN!

Author here. A friend mentioned this was on the front page so I wanted to stop by and make explicit that this advice is OUT OF DATE as far as I'm concerned. It's way too much hassle (I work in a much larger team now than I did then!) and doesn't play well with rebase etc.

These days I either recommend the jupytext approach (not tried it but seems sensible) or personally I just use Sphinx-gallery.

Advantages:

* Plain python files play well with IDE refactoring, Black formatter, etc etc.

* You now have a readymade 'tutorial' page for your docs.

* Files are run with every docs build, so you can configure things to alert you when they're broken.

Disadvantages:

* You end up editing a throwaway notebook file. If you forget to copy-paste your edits back to the source, and rebuild, you have lost your edits. However, this forces me to keep the 'temporary, exploratory' nature at the front of my mind and not allow the notebook code to grow too large before performing some clean-up.

Post reply on HN