Live data from Hacker News

What’s wrong with computational notebooks?

web.eecs.utk.edu

211–220 of 223 posts

Re: What’s wrong with computational notebooks?

#211

Earlier quoted context omitted.

Yep, the real complaint is “dead state”, not out of order execution. Worrying about linear flow per se turns out to be misguided based on lack of imagination for/experience with a better model: reactive re-rendering of dependent cells. Observable entirely solves the dead state problem, in a much more effective way than just guaranteeing linear flow would do. * * * More generally, Observable solves or at least amelior…

You can get a jupyter extension[1] that allows you to add tags and dependencies and this way construct the dependency graph as you go along. Of course, you have to do it manually and the interface is a bit clunky, but it does what it says. In practice I think taking care not to accidentally shadow variables is much more important: this dependency business only makes sense once you have a clear idea of what you need a…

I don’t understand what you are trying to say in your second paragraph, but I highly recommend you spend a few weeks playing with http://observablehq.com instead of speculating about the differences.

In practice, I find it to be dramatically better than previous notebook environments for data analysis, exploratory programming / computational research, prototyping, data visualization, and writing/reading interactive documents (blog posts, software library documentation, expository papers ...). It has a lower barrier to starting new projects, a lower-friction flow throughout

I find it better at every stage of my thinking process from blank page up through final code/document, and would recommend it vs. Jupyter or Matlab or Mathematica in every case unless some specific software library is needed which is unavailable in Javascript. The only other tool I really need is pen and paper, though I also use http://desmos.com/calculator and Photoshop a fair bit.

Re: What’s wrong with computational notebooks?

#212
post #181

Earlier quoted context omitted.

It's not even free software.

Developers need to pay bills somehow.

"Free software" is about freedom, not price. Putting the research that is your life's work inside of proprietary software that can be taken away from you, forever, at any time — that seems foolish.

Re: What’s wrong with computational notebooks?

#213
post #212
post #181

Earlier quoted context omitted.

Developers need to pay bills somehow.

"Free software" is about freedom, not price. Putting the research that is your life's work inside of proprietary software that can be taken away from you, forever, at any time — that seems foolish.

Copyleft derived "free software" can also be taken away from me.

Re: What’s wrong with computational notebooks?

#214
post #213
post #212

Earlier quoted context omitted.

"Free software" is about freedom, not price. Putting the research that is your life's work inside of proprietary software that can be taken away from you, forever, at any time — that seems foolish.

Copyleft derived "free software" can also be taken away from me.

Only if you violate the license.

By the way, I'd be interested in your thoughts on https://news.ycombinator.com/item?id=22083468.

Re: What’s wrong with computational notebooks?

#215

Earlier quoted context omitted.

Think of all the damage caused by excel. We replaced one set of avoidable catastrophes for another. But this time there’s no shame.

I'm a computational biologist and Excel has been the bane of my existence for 20 years. We've "known better" for all of that time, but I still deal with people passing around Excel files of data or having common spreadsheets on shared drives (or now Dropbox shared). We all "know better", but Excel is often the first thing that people try to keep track of data, and once a system works, there is just too much inertia t…

Are you hiring for summer positions?

Re: What’s wrong with computational notebooks?

#216
post #184

Earlier quoted context omitted.

I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…

>to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. My background is programming (instead of data analysis & modeling) so I'm sympathetic to your idealistic "software engineering" view... but I'm also sympathetic to the academics' side as explained by Yihui…

We shouldn't assume it will always exist. It exists because programming languages and tools are not as usable as they can be. That is something we can and should expect to change.

Re: What’s wrong with computational notebooks?

#217
post #187

I used Mathematica’s notebook interface quite heavily 15-20 years ago; Jupyter’s interface is a clone of that in many ways. At the time, my workflow was to use two different notebooks for everything: foo.nb and foo-scratch.nb. I’d get things working a piece at a time in foo-scratch.nb, not caring at all how it looked, not having to worry about leaving extra output or dead ends of explorations lying around; then the r…

I make a "scratch pad" section of my notebook and work on ideas there. Then once I've pieced together a function line by line and tested it a bit I move it up to where it should be in the chronological order of the notebook. Kind of like your two notebook system but makes copying easier in Jupyter.

I do the same, though it feels dangerous because both the good-copy and scratch sections share the same kernel. JupyterLab works on .ipynb files, and makes it way easier to copy (or drag and drop) cells between different notebooks. One of these days, I plan to switch to JupyterLab to get a sense of what else it offers above Jupyter Notebook.

Re: What’s wrong with computational notebooks?

#219

Earlier quoted context omitted.

'Thinking about the data' most often requires looking at the data from hundreds of different angles, quickly investigating its properties and statistics, maybe plotting or fitting a few things, checking some hypotheses etc (all of the above code you will most likely throw out after the initial stage). Same with the results - once you've coded something (perhaps outside of a notebook environment) and obtained results,…

Maybe you use a notebook I'm completely unfamiliar with, but my experience is that they allow you to write code, run it, and save the results in cells. My IDE does all of that except the saving of partial results part, but this can be done easily by just dumping your precomputed data to disk if you can't recompute it easily. In either case, an IDE gives you get an actual debugger, plus with IntelliJ it has a great da…

I use both PyCharm and JupyterLab on daily basis, typically dealing with multi-gb datasets.

If I'm writing a library or adding new features to one, or writing tests I'll use PyCharm sure thing, otherwise the notebook is a quicker way to sketch prototypes and always have a kernel with preloaded datasets and pre-imported stuff ready at hand. I don't want to wait 10 minutes to just load the data every time I want to check if my new function works well on it at big scale. That's one of the most important bits.

PyCharm is a clear winner at actually writing code that you won't throw in the bin 10 min later, and once you know what to write.

Debugging? Don't remember ever using PyCharm despite the fact it exists... either pudb or python-devtools or something else. I'd just write tests and things start working in the process. And btw you have pdb debugger (some weak version of it) in jupyter if you really need it. Docstrings? Press tab twice in the notebook. Or keep PyCharm open on the side so you can cmd-b. Profiling? Never a pycharm builtin, maybe something like flamegraph but an external tool anyway.

Re: What’s wrong with computational notebooks?

#220
post #5

I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of peopl…

That's part of what drove me to write TopShell, which is a notebook-like interface:

https://github.com/topshell-language/topshell

  * Information only flows downwards.
  * Computations are cached.
  * Things are automatically recomputed when the values they depend on change.
  * Things with effects is instead cleared and awaits confirmation before running.
Post reply on HN