Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git and Jupyter Notebooks Guide

reviewnb.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Git and Jupyter Notebooks Guide

#41

Post by cycomanic »

enriquto wrote: post #2

Curious that they discuss several options, but ignore the totally obvious one: just use jupytext [0]. Jupytext is a (tiny) jupyter extension that reads/writes notebooks as python files, with text cells being represented as comments. With jupytext, you do away with the stupid .ipynb format. As long as you don't need to save the cell outputs, which is the case for version control, jupytext is the way to go. People: pip…

This was the first thing I wanted to post when reading the article. Jupytext is excellent, although i typically use MyST (an extended Markdown syntax).

Re: Git and Jupyter Notebooks Guide

#42

Post by sfpotter »

tnecniv wrote: post #39
sfpotter wrote: post #16

Earlier quoted context omitted.

"unholy" "cleaner" "stateful" "bizarre" "bad taste" "hygiene" I'm not sure whether you're unaware or just feigning ignorance, but notebooks are frequently used to share partial results, often in the context of "research", however you may interpret it. Imagine a grad student or data scientist preparing some code and plots to show during a weekly meeting. In this context, the only thing that matters is quick progress a…

As someone who uses notebooks for research, outputs don’t play well with git. You can end up with very large commits that GitHub or wherever your repo lives may not like if you have a lot of plots and animations. Moreover, while research moves fast, reproducibility remains important. If your notebook is stateful, then when you share it I may not be able to recreate your result or you might have a bug due to something…

There are as many different "research workflows" as there are researchers.

I'm a researcher and don't use notebooks for all the reasons you outlined and more. I have my own approach to dealing with reproducibility which is low tech and works for me and my collaborators.

My comment is meant to point out that there are many researchers who view all of the problems you describe as unimportant and not worth spending time on.

Re: Git and Jupyter Notebooks Guide

#44

Post by tnecniv »

adament wrote: post #32
Zandikar wrote: post #14

Earlier quoted context omitted.

Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".

But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for loca…

Yeah if your analysis to takes hours to run, you should really split up the number crunching code and result analysis / visualization. Not only does it make version control of the code easier, you can save the output in an organized labeled manner (time-stamped, etc.) and, if you lose power or the kernel crashes, you don’t need to rerun the lengthy analysis if you want to make a change further down the pipeline.

Re: Git and Jupyter Notebooks Guide

#45

Post by tnecniv »

aqsalose wrote: post #35
pletnes wrote: post #10

Earlier quoted context omitted.

Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being co…

However, I wouldn't then use version control software like Git for versioning analysis objects, as it is designed for text file source control and diffs. (How one does a diff of a data object look like? If there is a natural text format to save it in, it still is usually quite messy, and Git doesn't really like Gb sized csvs.) My preferred workflow is to version the source files in Git and store the associated data o…

What’s your solution for hosting the archive of the data if you want to share it? That’s a weakness in my workflow.

Re: Git and Jupyter Notebooks Guide

#46

Post by kortex »

steve_gh wrote: post #34
kortex wrote: post #24

Earlier quoted context omitted.

It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case , which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field.…

Yeah but jupyter notebooks suck at providing reproducible data science. I encourage my team's not to use Jupyter for data science. Our preferred toolchain is based on make to build data science pipelines. Every step is scripted, and make ensures that upstream changes or script changes trigger downstream changes, ending with charting with gnuplot or similar. Our output charts all are not only timestamped but have a gi…

> Yeah but jupyter notebooks suck at providing reproducible data science.

Why? I have no problem with reproducibility when I use a little bit of discipline.

Your workflow does indeed sound nice but also sounds like it involves way more tooling and institutional knowledge. Anywhere I can learn more about it or see the scripts you use?

Re: Git and Jupyter Notebooks Guide

#47

Post by chthonicdaemon »

aqsalose wrote: post #35
pletnes wrote: post #10

Earlier quoted context omitted.

Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being co…

However, I wouldn't then use version control software like Git for versioning analysis objects, as it is designed for text file source control and diffs. (How one does a diff of a data object look like? If there is a natural text format to save it in, it still is usually quite messy, and Git doesn't really like Gb sized csvs.) My preferred workflow is to version the source files in Git and store the associated data o…

I use nbdime and have integrated it into my git diff so I see graphical diffs. Github renders notebooks with their output intact. They recently announced [rich notebook diffs](https://github.blog/changelog/2023-03-01-feature-preview-ric...) as well.

Re: Git and Jupyter Notebooks Guide

#48

Post by steve_gh »

kortex wrote: post #46
steve_gh wrote: post #34

Earlier quoted context omitted.

Yeah but jupyter notebooks suck at providing reproducible data science. I encourage my team's not to use Jupyter for data science. Our preferred toolchain is based on make to build data science pipelines. Every step is scripted, and make ensures that upstream changes or script changes trigger downstream changes, ending with charting with gnuplot or similar. Our output charts all are not only timestamped but have a gi…

> Yeah but jupyter notebooks suck at providing reproducible data science. Why? I have no problem with reproducibility when I use a little bit of discipline. Your workflow does indeed sound nice but also sounds like it involves way more tooling and institutional knowledge. Anywhere I can learn more about it or see the scripts you use?

I completely agree that a bit of discipline does wonders. But it is not just your discipline, it is the team's discipline.

What I need to ensure is that anyone picking up a piece of analysis 3 months later can reproduce exactly what was done. I've been burnt in the past by having to go back to the original analyst and be told "oh you run this bit of this notebook, then paste the results in over here, then run that". By insisting that everything is scripted and that there are no manual steps, we get a reproducible analytics pipeline.

The starting point for our methodology is the book "Guerilla Analytics" by Enda Ridge. It's worth reading.

Re: Git and Jupyter Notebooks Guide

#49

Post by Zandikar »

tnecniv wrote: post #44
adament wrote: post #32

Earlier quoted context omitted.

But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for loca…

Yeah if your analysis to takes hours to run, you should really split up the number crunching code and result analysis / visualization. Not only does it make version control of the code easier, you can save the output in an organized labeled manner (time-stamped, etc.) and, if you lose power or the kernel crashes, you don’t need to rerun the lengthy analysis if you want to make a change further down the pipeline.

It was an extreme example to drive home the point that one is "human scale time" and one is "computer scale time", people are reading far far too much into my choice of hours specifically there.
Post reply on HN