Making Git and Jupyter Notebooks play nice
timstaley.co.uk
Making Git and Jupyter Notebooks play nice
1–10 of 33 posts
Re: Making Git and Jupyter Notebooks play nice
#2I just clear all outputs before committing code. Works well enough for me, but it's cool to see a more advanced solution!
Re: Making Git and Jupyter Notebooks play nice
#3I just clear all outputs before committing code. Works well enough for me, but it's cool to see a more advanced solution!
The problem with that is when there are very long running cells and you want to see the output and keep developing, and committing code as you go along.
Re: Making Git and Jupyter Notebooks play nice
#4Things I’ve found in checked-in notebooks:
* database credentials.
* sensitive data
* a whole DataBricks webpage because the person didn’t understand how to export just the notebook.
* collections of notebooks named only what step in the process they are, and literally nothing about what they actually do.
* Whole base64 encoded images and zip files
* packages imported by manually manipulating system environment paths
* multi-processing/multithreading by shelling out and calling new python instances
* good old “don’t run these cells”
Re: Making Git and Jupyter Notebooks play nice
#5Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells
Re: Making Git and Jupyter Notebooks play nice
#6Things I’ve found in checked-in notebooks: * database credentials. * sensitive data * a whole DataBricks webpage because the person didn’t understand how to export just the notebook. * collections of notebooks named only what step in the process they are, and literally nothing about what they actually do. * Whole base64 encoded images and zip files * packages imported by manually manipulating system environment paths…
Untitled9.ipynb
Re: Making Git and Jupyter Notebooks play nice
#7Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells
I guess you write hooks then, you don't convert on every commit?
Re: Making Git and Jupyter Notebooks play nice
#8Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells
I guess you write hooks then, you don't convert on every commit?
Jupytext works as an extension in Jupyter. Your work is saved as py automatically whenever you work on the notebook. So it's not a conversion at commit time, unlike say nbstripout
Re: Making Git and Jupyter Notebooks play nice
#9Note: This article is from February 2017.
Re: Making Git and Jupyter Notebooks play nice
#10Or just use jupytext and only commit the .py files. Works for us. Commits just look like normal python code, with a few comment markers for cells
Thank you for mentioning jupytext. I had read about it but had forgotten the name. https://github.com/mwouts/jupytext