191–200 of 308 posts
Having spent a decent amount of time learning to be a programmer while doing scientific image analysis in Matlab (shudders from the real programmers), and with a decent amount of time spent in Mathematica as well, I just can't seem to buy into the Jupyter/notebook based programming enthusiasm. The talk linked in the article explains it better than I ever could, but for me, when I am leaving data in memory, it is much…
Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/
Version control for Jupyter notebooks was one of the biggest complaint I had. Specifically, diff and merge with the JSON files (.ipynb) is ugly. I built ReviewNb[1] to solve one of those problems (diff). Note that, there is nbdime[2] which works well for local diff/merge. The idea for ReviewNb is to have much tighter integration with GitHub etc. [1] https://reviewnb.com [2] https://nbdime.readthedocs.io/en/latest/
I think they fundamentally json is just the wrong format for these files. Speaking from (ancient and limited) experience I made a little notebook-style interpreter for learning scala back in 2009 or so called scalide. It saved its files ("scalapads") to XML. XML actually worked better in some ways since most of the code could live between the tags unescaped (sans &) so it merged / diffed the user code well. The meta-…
mynotebook.py
### (cell boundary)
"""Top-level unused strings (docstring-esque) rendered as markdown"""
def add(x, y):
return x + y
# jupyter-output-hash: 0123abc (which would link to some external key-value storage for the project)
As long as the output-hash storage is treated as append-only and is highly available (output cells could even be encrypted for security if this was a public cloud service, or you could even use a local or shared filesystem), then this file would not only parse and run as a perfectly valid Python file, but it would also hold references to outputs in a source-control friendly way. IDEs could show the cell outputs inline. If you rerun your notebook and get different outputs for some reason, `git diff` tells you exactly where things changed without being too messy. Basically, put outputs in off-chain storage, and just be a literate code file.
I feel like this would address most people's needs, no?
The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…
You parting "Granted..." is precisely what fills me with dread when I see notebooks. Yes, I have seen poorly done source files. I made more than a few myself. However, many of the practices we have grown into as sound programming advice seem to be largely thrown out the window for these notebooks. The irony, to me, is that I actually typically argue for the mixing of presentation and content. But to me, notebooks loo…
They probably didn't take off to the same extent as Jupyter because they're not free. IIRC MATLAB was quite expensive, particularly if you wanted to do anything specialised.
Earlier quoted context omitted.
The hard part is that introducing a tool like git (which requires you to choose moments to take a snapshot of the file, and then add some commit message) breaks the flow of interactive experimentation that notebooks are so good for. And then we need to find a way to make those commits useful, because the time ordering of commits could be different from the time order in which cells were run! That is what is crucial t…
I'm not a Jupyter user, but I solve the reproducibility problem with Make. As a project moves from exploration toward production, the entire thing is wrapped into a Makefile that can flow from raw data to publication in a single call to make.
https://gist.github.com/ontouchstart/854a3c280b81f530d3ae9cb...
The notebook generated by nbconvert (see the instruction in the Makefile) is too big to display in GitHub gist Web UI but works fine in nbviewer.
https://nbviewer.jupyter.org/gist/ontouchstart/854a3c280b81f...
Having spent a decent amount of time learning to be a programmer while doing scientific image analysis in Matlab (shudders from the real programmers), and with a decent amount of time spent in Mathematica as well, I just can't seem to buy into the Jupyter/notebook based programming enthusiasm. The talk linked in the article explains it better than I ever could, but for me, when I am leaving data in memory, it is much…
Which talk? I didn't see any link to a video.
https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...
https://www.youtube.com/watch?v=7jiPeIFXb6U&feature=youtu.be
Earlier quoted context omitted.
I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).
I'm coming to realize one of the key skills for a data engineer to have nowadays is "productionizing" notebook code from data scientists and PMs and teaching them to make it more testable and modular in the first place.
The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…
The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…
> When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. At the core of this, as some others may have already alluded to already, is that many academic scientists have not been socialized to make a distinction between development and pro…
Geologist here - definitely true in my field. Nonetheless, while I don't develop in notebooks at all, I do use them for "reproducibility" in a sense -- by putting a bit of dependency info in a github repo along with a .ipynb file, I can do things like this: https://mybinder.org/v2/gh/brenhinkeller/Chron.jl/master?fil...
Which ends up being useful when a lot of folks in my field don't do any computational work at all, so being able to just click on a link and have something work in browser is a big help.
Does anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soo…