31–40 of 49 posts
If you don't need to "commit" the output, you can just use nbconvert[0]: jupyter nbconvert --clear-output --inplace my_notebook.ipynb So you can use git as usual, like for code. [0] -- https://nbconvert.readthedocs.io/en/latest/
Earlier quoted context omitted.
Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being co…
Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".
I haven't read the link and I'm not going to. I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code. To me that killed jupyter notebooks. Why use something that is strictly…
It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case , which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field.…
> Nothing even comes close. There's a reason it's dominant in the data science field.
> Your workflow works for you but the jupyter workflow works for millions of students, data scientists, and even developers. Heck I even know all the ways to avoid jupyter, and I still use it often, because it's so convenient.
Copy pasting your comment here so when you eventually delete it people can still see the ignorance.
You have absolutely no clue what you're talking about. Worse, it seems like you didn't read what you're responding to.
I haven't read the link and I'm not going to. I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code. To me that killed jupyter notebooks. Why use something that is strictly…
It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case , which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field.…
Our preferred toolchain is based on make to build data science pipelines. Every step is scripted, and make ensures that upstream changes or script changes trigger downstream changes, ending with charting with gnuplot or similar. Our output charts all are not only timestamped but have a git commit id. And our source repositories contain a data manifest so we have commit IDs right into ETL stages into the DB.
End result is that in a couple of months, when the CxOb asks about some piece of work and pulls out a chart, we can trace the entire data pipeline used to create it, and reproduce it if required. That saves so much hassle!
Earlier quoted context omitted.
Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or…
Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being co…
(How one does a diff of a data object look like? If there is a natural text format to save it in, it still is usually quite messy, and Git doesn't really like Gb sized csvs.)
My preferred workflow is to version the source files in Git and store the associated data objects in a separate archive directory with meaningful name and the hash of commit of generating code as metadata attribute.
Now if you had a version control "IDE" software that would render changes in figures and other blobs nicely, then it would make sense to build a workflow around it.
Earlier quoted context omitted.
It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case , which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field.…
Yeah but jupyter notebooks suck at providing reproducible data science. I encourage my team's not to use Jupyter for data science. Our preferred toolchain is based on make to build data science pipelines. Every step is scripted, and make ensures that upstream changes or script changes trigger downstream changes, ending with charting with gnuplot or similar. Our output charts all are not only timestamped but have a gi…
That depends on how you use the notebooks.
With just a tiny bit of discipline, you can integrate notebook users into your sane workflow. For example, encourage people to restart the kernel and run all cells a few times per day (and definitely, before sharing anything). Meaningful output artifacts can be saved into files, that are later read by the notebook and displayed.
Then, when users are satisfied with their notebook, they save it as a python file thanks to jupytext, and commit it to git.
This workflow integrates well with your makefile setup: to reproduce the notebook and obtain its results you simply run it as a script. If you want a pdf or a static html that shows the notebook as-is, you can nbconvert it from your makefile.
For example, if your makefile has lines like these:
%.ipynb : %.py ; jupytext $Notice that you can simply run "python foo.py" as well, to produce the valuable output artifacts.
In the end, jupyter becomes just an editor of python files. A fancy editor, that allows interactive execution of pieces of code, which is great.
I haven't read the link and I'm not going to. I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code. To me that killed jupyter notebooks. Why use something that is strictly…
It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case , which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field.…
Earlier quoted context omitted.
Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or…
"unholy" "cleaner" "stateful" "bizarre" "bad taste" "hygiene" I'm not sure whether you're unaware or just feigning ignorance, but notebooks are frequently used to share partial results, often in the context of "research", however you may interpret it. Imagine a grad student or data scientist preparing some code and plots to show during a weekly meeting. In this context, the only thing that matters is quick progress a…
Earlier quoted context omitted.
Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or…
"unholy" "cleaner" "stateful" "bizarre" "bad taste" "hygiene" I'm not sure whether you're unaware or just feigning ignorance, but notebooks are frequently used to share partial results, often in the context of "research", however you may interpret it. Imagine a grad student or data scientist preparing some code and plots to show during a weekly meeting. In this context, the only thing that matters is quick progress a…
Moreover, while research moves fast, reproducibility remains important. If your notebook is stateful, then when you share it I may not be able to recreate your result or you might have a bug due to something lingering in the notebook state. Having your outputs is convenient, but if I download the notebook, run it myself, and find that the code doesn’t run because there’s some variable that got defined earlier in your session but that code got deleted during iteration, that’s really not helpful. It’s the equivalent of handing someone your lab notebook but you kept erasing over early pages to make room for new content.
That’s one example of a bug. You could easily introduce more subtle bugs where the state leads to invalid results.