21–30 of 49 posts
I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code.
To me that killed jupyter notebooks. Why use something that is strictly worse in every respect?
The native GitHub feature in preview will make this a lot better for those able to use it https://github.blog/changelog/2023-03-01-feature-preview-ric...
I definitely do like to strip notebooks and make them run-idempotent to the best of my ability, but sometimes you just need stateful notebooks. And since .ipynb are technically json but in reality act more like a binary file format (with respect to diffing), DVC is the ideal tool to store them. Don't get me started on git annex or LFS, both of those took years off my life due to stress of using them and them bugging out.
Also I am hardly a fan of XML, but does anyone feel like notebook files would have been a near-ideal use-case of it? It's literally a collection of markup. The fact that json was chosen over xml I think is somewhat damning of xml as an application data storage format. I think xml is perfectly cromulent as a write-once-read-many presentation format or rendering target (html, svg, GeniCam api info), but it seems to flounder in virtually every other domain it's been shoehorned into, with the exception of office application formats.
Actually, downthread there is a link to a jupyer enhancement proposal for a .nb.md markdown based format. I think this is great. One theme I keep coming across in my computer science journey is that formats which have mandatory closing endcaps are kind of a PITA. It seems the stream-of-containers (with state machines as needed) is all-around better. JSON-LD is better than JSON, streaming video formats are better than ones that stick metadata at the end, zip is... an eldritch horror, etc.
I haven't read the link and I'm not going to. I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code. To me that killed jupyter notebooks. Why use something that is strictly…
Nothing even comes close. There's a reason it's dominant in the data science field.
Your workflow works for you but the jupyter workflow works for millions of students, data scientists, and even developers. Heck I even know all the ways to avoid jupyter, and I still use it often, because it's so convenient.
Earlier quoted context omitted.
Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or…
Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being co…
Curious that they discuss several options, but ignore the totally obvious one: just use jupytext [0]. Jupytext is a (tiny) jupyter extension that reads/writes notebooks as python files, with text cells being represented as comments. With jupytext, you do away with the stupid .ipynb format. As long as you don't need to save the cell outputs, which is the case for version control, jupytext is the way to go. People: pip…
What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versioning.
Earlier quoted context omitted.
Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or…
"unholy" "cleaner" "stateful" "bizarre" "bad taste" "hygiene" I'm not sure whether you're unaware or just feigning ignorance, but notebooks are frequently used to share partial results, often in the context of "research", however you may interpret it. Imagine a grad student or data scientist preparing some code and plots to show during a weekly meeting. In this context, the only thing that matters is quick progress a…
As a concrete example, this one-liner of Python code is much more interesting to those who don't recognize it when it's presented with the associated output.
4*sum([(random.random()**2 + random.random()**2)**.5 (The one-linear above is a monte-carlo simulation which approximates Pi. On one run, this result came to 3.1410416.)
Earlier quoted context omitted.
What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versioning.
I use Jupytext since years. It allows me to have three types of synced notebook versions: 1) .ipynb (for opening/running), 2) .md (formatted code+comments, without outputs) and 3) *.py (python formatted, code+comments). I commit the Markdown-version, but I also use the py-version of notebooks for chained notebook imports. Allows me to split larger notebooks into multiple smaller ones. Both of these options are a bles…
I haven't read the link and I'm not going to. I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code. To me that killed jupyter notebooks. Why use something that is strictly…
It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case , which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field.…
I wish that non-emacs implementations of org were more commonplace, as it's a pretty sane markup language and supports embedded code and graphics, diffs nicely, and doesn't introduce the insanity of JSON.