Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are there any good Diff tools for Jupyter Notebooks?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Ask HN: Are there any good Diff tools for Jupyter Notebooks?

#22

Post by Royi »

nolroz wrote: post #7

Visual studio code has a diffing view for notebooks that looks very promising. https://code.visualstudio.com/docs/datascience/jupyter-noteb...

I wish for a simple option in VS Code: On close of a Jupyter Notebook clear its output. Or something that separate the display of the output from the saved file (Still `ipnyb` file). See [1].

[1]: https://github.com/microsoft/vscode-jupyter/issues/9514

Re: Ask HN: Are there any good Diff tools for Jupyter Notebooks?

#25

Post by dahart »

aulin wrote: post #20
dahart wrote: post #6

Can you talk more about why you’re working in Jupyter Notebooks at a level that needs diff reviews? Are you reviewing your own work, or the work of others? One option would be to start a policy to always “restart and clear output” before saving. This cleans the output cells and makes the .ipynb files diffable. Just happens to also make them nice for storing in version control. Another option would be to work in pure…

Not OP but restart and clear output can be quite compute intensive if you're working with big datasets or training ML models. There are many ways to mitigate this like saving weights and only redo the inference but it's not always worth it when you're iterating through models and parameters or doing exploratory data analysis. Most of the time you want to just keep results/outputs of previous run and improve from ther…

That’s a great point, I sometimes avoid clearing outputs when I’m playing with Pytorch just because retraining takes a while. This has been motivating me to learn how to be fluent with saving weights to disk.

Re: Ask HN: Are there any good Diff tools for Jupyter Notebooks?

#27

Post by ivansavz »

cschmidt wrote: post #2

There is https://nbdime.readthedocs.io/en/latest/ , although I haven't used it personally to know how good it is.

I use nbdime daily for notebooks under version control thanks to the following configs in my ~/.gitconfig (global git config).

   [diff "jupyternotebook"]
     command = git-nbdiffdriver diff --ignore-details
   [difftool "nbdime"]
     cmd = git-nbdifftool diff --ignore-details \"$LOCAL\" \"$REMOTE\" \"$BASE\"
I'm not sure if this is a standard setup or if I copy-pasted from some blog post, but overall it's working great.

There are some issues with it, like (1) will unnecessarily mark graphics as changed (e.g. re-generated figures from the same code), and (2) the diffs become less meaningful if large chunks of cells were moved, but overall it works great.

If it supported a `--color-words` option then it would be super helpful for seeing only which words have changes, instead of whole lines changed (very good for long paragraphs of Markdown text).

Re: Ask HN: Are there any good Diff tools for Jupyter Notebooks?

#28

Post by domenicrosati »

dahart wrote: post #19
pen2l wrote: post #15

Earlier quoted context omitted.

edit: Googling reveals nbdime, has this been looked into? - https://nbdime.readthedocs.io/en/latest/ Not OP but I can imagine easily the need for what he's asking. You'll find a lot of algorithms for data and image processing saved as notebooks these days offered to you. Let's say you make some changes from the provided code and after a handful of changes something is not working right. You might want to diff from wh…

Oh I can totally imagine use-cases too, but I’d love to hear what the OP’s use case actually is. I also agree completely on the disruption that Jupyter brings, and that it has just massive benefits. But when a workflow isn’t giving you everything you want, it’s worth evaluating whether the tools you’re using are the right tools for the job, right? One example would be that Jupyter is well designed for a lot of protot…

Hey there - OP here. I haven't used spyder I'll have it check it out.

The primary use case is: I am a researcher in nlp where speed of prototyping is key. I work in an environment where research fragments are primarily jupyter notebooks. So needing to diff notebooks is typically reviewing my own changes when modifying my and others research sketches. Since its helpful to see how code changes.

What really resonates with me is what others have said which is I need to run cells that take 2-6 hours to compute so recomputing cells is annoying... I dont love notebooks for their messy state which cause obvious problems that are very annoying.. and I am not an advocate for notebooks for production for this reason but the flexibility of computing stuff and having that persist and doing downstream prototyping makes notebooks amazing! Markdown and latex in there is also really helpful.

The secondary use case is PRs but... typically reviewing others research code isnt at the granular level of notebook riffs across a few commits so it deosnt come up often.

Re: Ask HN: Are there any good Diff tools for Jupyter Notebooks?

#29

Post by ivansavz »

pen2l wrote: post #15
dahart wrote: post #6

Can you talk more about why you’re working in Jupyter Notebooks at a level that needs diff reviews? Are you reviewing your own work, or the work of others? One option would be to start a policy to always “restart and clear output” before saving. This cleans the output cells and makes the .ipynb files diffable. Just happens to also make them nice for storing in version control. Another option would be to work in pure…

edit: Googling reveals nbdime, has this been looked into? - https://nbdime.readthedocs.io/en/latest/ Not OP but I can imagine easily the need for what he's asking. You'll find a lot of algorithms for data and image processing saved as notebooks these days offered to you. Let's say you make some changes from the provided code and after a handful of changes something is not working right. You might want to diff from wh…

> https://jupyterlab.readthedocs.io/en/stable/user/rtc.html

Wow! Realtime notebook collaborative editing! This is going to be so cool for teaching (allow students to fill-in part of the code block).

Have you tried this yes? Is the idea to run jupyter on a machine with a public IP and port 8888 open allowing the server to be accessed from multiple people at the same time? Would this work services like `ngrok` that make you personal computer available online?

Re: Ask HN: Are there any good Diff tools for Jupyter Notebooks?

#30

Post by amirathi »

Here are tools people commonly use for notebook version control with git -

[1] nbdime to view local diffs & merge changes

[2] jupytext for 2-way sync between notebook & markdown/scripts

[3] JupyterLab git extension for git clone / pull / push & see visual diffs

[4] Jupyerlab gitplus to create GitHub PRs from JupyterLab

[5] ReviewNB for reviewing & diff'ing notebook PRs / Commits on GitHub

Disclaimer: While I’m the author of last two (GitPlus & ReviewNB), I’ve represented the overall landscape in an unbiased way. I've been working on this specific problem for 3+ years & regularly talk to teams who use GitHub with notebooks.

[1] https://nbdime.readthedocs.io

[2] https://jupytext.readthedocs.io

[3] https://github.com/jupyterlab/jupyterlab-git

[4] https://github.com/ReviewNB/jupyterlab-gitplus

[5] https://www.reviewnb.com/

Post reply on HN