What are the currently available CI options for notebooks? You'd think this would be one of the first tools people would need to make sure notebooks are reproducible, but there seems to be little sign of CI usage.
What’s wrong with computational notebooks?
91–100 of 223 posts
Re: What’s wrong with computational notebooks?
#92Earlier quoted context omitted.
I just happened to be reading Peter Naur's "Programming as theory building" recently. It strikes me that taking its theme even a little seriously helps understand why notebooks are so popular. Notebooks happen to be convenient tools for exploring a new domain (interactively). Irrespective of how much software purists might complain, conventional software engineering provides very few tools/solutions/practices for tha…
I just refactor into a module that I import into my notebook as I go along. This lets me use the notebook for quick prototyping, but also productionize faster if need be.
That only works after the code in the module is largely "frozen". It doesn't work well if you're experimenting with ideas inside the module. OTOH, if the algorithm is largely frozen, and you're trying to experiment with its performance on a bunch of examples, the workflow of putting the algorithm in a module and using a notebook to interface with data and visualize results is quite useful.
That is basically what I meant by knowing when to transition from one mode to the other.
Here's a concrete example (maybe somebody considers this an inspiring challenge?), to illustrate how notebooks are infuriating in their primitiveness, but still better compared to using an editor on source files: Imagine a beginner trying to write/learn a sorting algorithm, and who would like to keep experimenting with their code and observing what happens on examples, possibly profiling space/time complexity along the way.
To expand on my point above, there are actually three distinct computational use cases, not just two: Interactive learning -> Sharing insights with others -> Productionizing code.
Re: What’s wrong with computational notebooks?
#93I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of peopl…
A couple years out of college we finally took a hard look at the credit cards and realized we had fucked up. We were gonna buckle down, pay the cards down hard for a while, 'color' our money so we both had discretionary spending separate from, say, the power bill. She had much more Excel experience than I did so she worked up a spreadsheet. It was bad. We had worked up some 'fair' notion of proportionality and she ba…
The entire accounting department at any company exists to make sure their numbers are spot on. If your wife had an entire accounting department scrutinizing her numbers, they'd find the discrepancy. These are people who were willing to sacrifice their entire professional career and their lives during busy season at least to do nothing but tinker with excel for 40+ years; always trust a masochist verging on the insane.
Re: What’s wrong with computational notebooks?
#94Earlier quoted context omitted.
Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.
as a counterpoint, plenty of R folks are pretty happy doing all of that in Rstudio
It is interesting to see this discussion about notebooks while I'm thinking about all the RStudio users who do all their work inside the IDE and are pretty happy. Notebooks seem like such an inferior tool to me. I'm also extremely bias.
Re: What’s wrong with computational notebooks?
#95I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of peopl…
Be careful what you wish for. Hot reloads can become very expensive. Especially when it comes to computationally heavy tasks that notebooks are built for. If you decide you want hot reloads by default, it'd mean each time you click on a cell and then click on another you'd be restarting the whole notebook. If you had massive datasets you were loading or other args that you were parsing manually or at prompt, you'd ha…
I don't think re executing by default would be beneficial. I just don't want state present in the interpreter that isn't in any live cell, and I only want time to flow one direction. Other than that, I think the other ergonomics of notebooks are fine.
Re: What’s wrong with computational notebooks?
#96Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
> Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
It's not clear who the audience is. It sounds like most people who complain about them are software people and not researchers/scientists.
For someone like me, who once did computational research using MATLAB, and later analyzed data for my job, Jupyter is not worse, and is in most ways superior. Let's take your points one by one:
> Participants stated they often downloaded data outside of the notebook from various data sources since interfacing with them programmatically was too much hassle.
This was the norm with MATLAB, Excel and JMP as well, unless someone wrote code to autodownload (extremely rare - less than 1% of people did that). And if you are going to write code to get the data from somewhere, it's much nicer in Jupyter than in these other tools.
> Not only that, but notebooks often crash with large data sets (possibly due to the notebooks running in a web browser).
I honestly have not seen this, and the reason makes no sense. Your browser is not handling the data. The kernel is. I mean yes, if you try to load several GB of data in pandas, it's possible you will have problems if you run out of RAM, but this has nothing to do with notebooks.
> Once the data is loaded, it then has to be cleaned, which participants complained is a repetitive and time consuming task
This was as much a problem prior to notebooks as it is now. Notebooks did not make this any worse.
> Explore and analyze. Modeling and visualizing data are common tasks but can become frustrating. For example, we observed one participant tweak the parameters of a plot more than 20 times in less than 5 minutes.
It was even worse with MATLAB. Ditto for Excel. JMP is a bit nicer for visualization, though.
> Notebooks do not have all of the features of an IDE, like integrated documentation or sophisticated autocomplete, so participants often switch back and forth between an IDE (e.g., VS Code) and their notebook.
It may be better now, but this was a problem in MATLAB as well.
> While it is easy to share the notebook file, it is often not easy to share the data.
This is as true with MATLAB, JMP, etc. A lot of the complaints about it being hard to reuse notebooks is because notebooks at least attempt to be reproducible, and thus many more people attempt it. Prior to notebooks, I know almost no one who tried to share MATLAB analyses, because it was such a pain to do so.
> Notebooks as products. If a large data set is used, as one might expect in production, then the notebook will lose the interactivity while it is executing. Also, notebooks encourage "quick and dirty" code that may require rewriting before it is production quality.
I suppose some people are trying to make products out of notebooks, and this is where all the recent grief I see is coming from. I do not think it was the primary goal of notebooks, though. They were meant for data analyses and prototyping, not for production use.
Re: What’s wrong with computational notebooks?
#97Earlier quoted context omitted.
I just refactor into a module that I import into my notebook as I go along. This lets me use the notebook for quick prototyping, but also productionize faster if need be.
That only works after the code in the module is largely "frozen". It doesn't work well if you're experimenting with ideas inside the module. OTOH, if the algorithm is largely frozen, and you're trying to experiment with its performance on a bunch of examples, the workflow of putting the algorithm in a module and using a notebook to interface with data and visualize results is quite useful. That is basically what I me…
Why doesn't work with experimenting with the module? In jupyter, if you're using auto reload then the module will refresh every time you use it.
Re: What’s wrong with computational notebooks?
#98(A frequent Jupyter Notebook user here. For data exploration, and teaching deep learning - then Colab is indispensable.) The main question is: what are the alternatives, for data exploration (and sharing its results). Similarly, for data science tool demos, Notebooks shine. IMHO the problem is not in the notebooks, but in how they are being used (i.e. the workflow). By writing scripts in py files, and using notebooks…
> The only build-in problem with Jupyter Notebooks is JSON, mixing input and output (and making it pain to work with version control)
Absolutely. JSON notebook format makes it very hard to do code reviews, merge in remote changes etc. After being frustrated with lack of solutions, I built ReviewNB[1] specifically to do notebook code reviews on GitHub. Alternatively, Nbdime[2] is also a nice open source library to see diffs locally & merge in changes.
Re: What’s wrong with computational notebooks?
#99I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.
Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.
That kind of depends on your process. In many cases pdb (or the debugging interface in your IDE of choice) works just fine for that. It's certainly not "terrible".
After the exploration and preprocessing stage I personally don't see much benefit of the notebook model, training/evaluation and any meaningful visualization takes forever anyway, that means I need to cache and persist intermittent results. With that it doesn't really matter all too much if I work on it in vim&pdb, an IDE, or Jupyter.
Re: What’s wrong with computational notebooks?
#100I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of peopl…
My problem with notebooks is that I feel like the natural mental model for them is a spreadsheet mental model, not a REPL mental model. Under that assumption, changing a calculation in the middle means that all of the cells that depend on that calculation would be updated, but instead you need to go and manually re-run the cells after it that depend on that calculation (or re-run the entire notebook) to see the effec…
I keep seeing the complaint about state and I am getting sick of it. JUST HIT "RESTART KERNEL AND RERUN ALL CELLS," HOW HARD CAN IT BE?