Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Git and Jupyter Notebooks play nice

timstaley.co.uk

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Making Git and Jupyter Notebooks play nice

#21

Post by SifJar »

leni536 wrote: post #19
nabdab wrote: post #18

Earlier quoted context omitted.

I love jupytext, but i feel like it’s a patch on a problem that should have just been solved. Just change jupyter to work directly in the genereres file format and skip the “pair files” hassle.

Or it could be solved at the git tooling side by introducing Jupyter specific merge and diff tools.

adding git tooling for a specific file type seems like a slippery slope, no? (assuming you are saying that git itself should have this tooling built in - if you mean some sort of addon, fair enough but then everyone who uses jupyter & git needs to install that addon)

Re: Making Git and Jupyter Notebooks play nice

#22

Post by leni536 »

SifJar wrote: post #21
leni536 wrote: post #19

Earlier quoted context omitted.

Or it could be solved at the git tooling side by introducing Jupyter specific merge and diff tools.

adding git tooling for a specific file type seems like a slippery slope, no? (assuming you are saying that git itself should have this tooling built in - if you mean some sort of addon, fair enough but then everyone who uses jupyter & git needs to install that addon)

I am not suggesting that git itself should ship with a bunch of custom merge utilities for specific file types. git ships with a mechanism that allows custom merge drivers. Setting up custom merge drivers might not be ergonomic right now, but it could have some benefits compared to the transformation approach. For example merge conflicts could result in a valid notebook and it could be manually resolved inside the notebook interface, no need to dive into the text file.

Re: Making Git and Jupyter Notebooks play nice

#23

Post by Tarq0n »

morotter wrote: post #14

I found a good compromise by using VSCode Python extension. You can import Jupyter notebooks as Python scripts and the other way around [0]. If I need to work on a notebook, I prefer working with a Python script and the interactive window [1]. Then I commit both script and ipynb version. [0] https://code.visualstudio.com/docs/python/jupyter-support-py... [1] https://code.visualstudio.com/docs/python/jupyter-support-p…

VSCode's solution is awesome. Rmarkdown works the same way, and it makes a lot of sense. Let code live as plaintext, it's native format, and only render results at runtime. Trying to preserve results in the file just makes things needlessly stateful.

Re: Making Git and Jupyter Notebooks play nice

#25

Post by stared »

I have some projects based on Jupyter Notebook (e.g. https://github.com/stared/thinking-in-tensors-writing-in-pyt...), and collaboration sucks. Even with git, I need to resort to "don't touch these files, I am working on them right now".

I was thinking about using RMarkdown files (vide https://towardsdatascience.com/version-control-with-jupyter-...) as for them diffs make sense.

Does any of you use this approach, or have insights on how to make it good for collaboration AND visible on GitHub?

Re: Making Git and Jupyter Notebooks play nice

#26

Post by lukasm »

TimSAstro wrote: post #20

Hi, HN! Author here. A friend mentioned this was on the front page so I wanted to stop by and make explicit that this advice is OUT OF DATE as far as I'm concerned. It's way too much hassle (I work in a much larger team now than I did then!) and doesn't play well with rebase etc. These days I either recommend the jupytext approach (not tried it but seems sensible) or personally I just use Sphinx-gallery. Advantages:…

Can you not use https://code.visualstudio.com/docs/python/jupyter-support-py... to export/import?

Re: Making Git and Jupyter Notebooks play nice

#28

Post by wodenokoto »

A lot of people mentioning what is essentially R markdown files as a better approach.

In .rmd the notebook is just markdown where code segments inside triple backticks can be executed.

That sounds nice and git compatible (and it is), but you lose a lot of the convenience of Jupiter notebook, namely that output isn’t stored together with input.

The nice thing about Jupyter notebooks is you get story, code and results in a single package.

Re: Making Git and Jupyter Notebooks play nice

#29

Post by stared »

wodenokoto wrote: post #28

A lot of people mentioning what is essentially R markdown files as a better approach. In .rmd the notebook is just markdown where code segments inside triple backticks can be executed. That sounds nice and git compatible (and it is), but you lose a lot of the convenience of Jupiter notebook, namely that output isn’t stored together with input. The nice thing about Jupyter notebooks is you get story, code and results…

Output stored with input is precisely the problem for version control. Not only committing generated content but generated content mixed with source code.

Re: Making Git and Jupyter Notebooks play nice

#30

Post by dracodoc »

wodenokoto wrote: post #28

A lot of people mentioning what is essentially R markdown files as a better approach. In .rmd the notebook is just markdown where code segments inside triple backticks can be executed. That sounds nice and git compatible (and it is), but you lose a lot of the convenience of Jupiter notebook, namely that output isn’t stored together with input. The nice thing about Jupyter notebooks is you get story, code and results…

With Rmarkdown you have 2 options, both have its own merits

1. save code and output separately, so you can save the output (but not comparing the output versions), or always generate output from scratch when needed (this actually help to ensure reproducibility)

2. or you can use R notebook format, which save the result with document together, in some companion folders.

Post reply on HN