Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What’s wrong with computational notebooks?

web.eecs.utk.edu

Post reply on HN
Print view View original

61–70 of 223 posts

Re: What’s wrong with computational notebooks?

#61

Post by bloaf »

So literally all of these complaints are about their particular implementations of notebooks, not the concept of computational notebooks in general, or are all computational notebooks destined to have unstable kernels?

In my mind, notebooks should be married to a functional style of programming, where you use the notebook's markup to thoroughly explain and document your functions. Below your "function definition" section, you keep a "trying things out" section where you actually plug the data into your functions for debugging/visualizations. You can't shoot yourself in the foot with variables because all the work is done in your function's lexical scope. You can shoot yourself in the foot with stale function definitions, but a good notebook interface gives you the ability to clear function definitions and run groups of cells, so you can make sure you always run your functions in a group that starts with a "clear function definitions" cell.

When you are done, you just cut the "trying things out" section into a second notebook which references the functions in the first and viola, you've got a very well documented library of functions, and a new work notebook where you can freely polish your visualizations/whatever.

Re: What’s wrong with computational notebooks?

#62

Post by rcar »

wwarner wrote: post #44

I have been heads down in jupyter for the past couple of weeks and I finally realized I just DO NOT LIKE IT AT ALL ! Cracks started appearing and then suddenly there was an avalanche of disappointment. The first crack -- it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. I imported all the TeX goodness, and it looked pretty nice, but I couldn't sh…

To offer some help for 2 and 4, you can get a script out of a notebook with jupyter nbconvert --to python, which you can even include as a cell that starts with ! since that will run shell commands.

For part of 3, there is global search and replace accessible either via Edit -> Find and Replace or Esc-F, and it includes case sensitivity toggles, regex search, and the option to change the current cell or globally.

For 1, I think there are some plugins that can make that tidier, but I've generally just accepted that if the people I'm presenting to don't want to see code then I'll just need to make a set of slides out of the whole thing.

Re: What’s wrong with computational notebooks?

#63

Post by ivirshup »

tgb wrote: post #47
sillysaurusx wrote: post #37

Earlier quoted context omitted.

It's best to think of the notebook as a REPL. So you'd want to run `del foo` on the old name. In fact, this is a good counterexample. Why should the notebook delete the old variable name? What if its value is a thread currently executing? Notebooks are REPLs, and it's better to get used to that than to try to enforce some confusing time traveling.

But, strangely, Jupyter doesn't also give you a REPL (like, say, R Studio does). I'm always making new cells in the middle to output the column names of my spreadsheet, and then I have to delete them. I used to just always have an ipython REPL running and test things out in there as I write. You can start a ipython instance on the same kernel but I found that messed up my plots when I did that IIRC.

You can get a REPL attached to a notebook in jupyter. When you open a console in jupyter-lab you have the option of attaching it to an already running kernel. Using the notebook interface you can connect a console using `jupyter console --existing`. By default this connects to the most recent session, but you can also specify a session by passing a token.

Re: What’s wrong with computational notebooks?

#64

Post by pottertheotter »

I use Jupyter Lab with Python every day. It's where I do my initial data exploration and cleaning. Jupyter Lab is not perfect, but most of these findings seem like they are more issues of inexperience with technology and programming, not computational notebooks.

Re: What’s wrong with computational notebooks?

#65

Post by IanCal »

sillysaurusx wrote: post #37
IanCal wrote: post #19

Earlier quoted context omitted.

If people are wondering about cases that can cause this - a common one (for me) is a mis-spelled variable name. If you go back and change it, the old one is still there and if you make the same mistake twice you will have code that runs but doesn't work. It's then really not obvious why it doesn't work.

It's best to think of the notebook as a REPL. So you'd want to run `del foo` on the old name. In fact, this is a good counterexample. Why should the notebook delete the old variable name? What if its value is a thread currently executing? Notebooks are REPLs, and it's better to get used to that than to try to enforce some confusing time traveling.

> It's best to think of the notebook as a REPL.

With sections of the history easily replaced.

> So you'd want to run `del foo` on the old name.

And then delete this line, because if you leave it in it'll break when you try and run the file all the way through.

> Notebooks are REPLs, and it's better to get used to that than to try to enforce some confusing time traveling.

Do you mean by treating them as append only and never rerunning any cells?

> In fact, this is a good counterexample. Why should the notebook delete the old variable name? What if its value is a thread currently executing?

The notebook has no idea if it can or can't, but that doesn't mean that leaving it in is good it's simply the only realistic option.

Re: What’s wrong with computational notebooks?

#66

Post by jpeloquin »

chirau wrote: post #43
moultano wrote: post #5

I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of peopl…

Be careful what you wish for. Hot reloads can become very expensive. Especially when it comes to computationally heavy tasks that notebooks are built for. If you decide you want hot reloads by default, it'd mean each time you click on a cell and then click on another you'd be restarting the whole notebook. If you had massive datasets you were loading or other args that you were parsing manually or at prompt, you'd ha…

> I think it is a good thing that notebooks separate instructions and re-execute manually by default. The cost of the alternative is just too high.

Maybe add a "lock" toggle so a user can block a cell from being automatically executed? The heavy numeric setup tasks could then be gathered in a few cells and locked, leaving the lighter plotting & summary stats cells free to update reactively.

Re: What’s wrong with computational notebooks?

#67

Post by IanCal »

antipaul wrote: post #50
sillysaurusx wrote: post #37

Earlier quoted context omitted.

It's best to think of the notebook as a REPL. So you'd want to run `del foo` on the old name. In fact, this is a good counterexample. Why should the notebook delete the old variable name? What if its value is a thread currently executing? Notebooks are REPLs, and it's better to get used to that than to try to enforce some confusing time traveling.

Agree. Get used to the habit of deleting old objects/names when you are replacing them, if you work in notebooks

It's an easy thing to miss though, because you also then need to delete the line of code you used to delete the old object/name so you have no record of cleaning up after yourself.

Re: What’s wrong with computational notebooks?

#69

Post by grp000 »

comment_guy wrote: post #59

I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.

Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.

Re: What’s wrong with computational notebooks?

#70

Post by kvlr »

moultano wrote: post #5

I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of peopl…

This is how runkit does it for nodejs and I think it’s working quite well for them.

We (at Nextjournal) tried doing the same for other languages (Python, Julia, R) and felt that it didn’t work nearly as well. You often want to change a cell at the top e.g. to add a new import and it can be quite annoying when long-running dependent cells re-execute automatically. I now think that automatic execution of dependent cells works great when your use case is fast executing cells (see observablehq) but we need to figure out something else for longer running cells. One idea that I haven’t tried yet is only run cells automatically that have executed within a given threshold.

I hear a lot of complaints about hidden state but I think it’s less of a problem in reality. It’s just a lot faster than always rerunning things from a clean slate. Clojure's live programming model [1] works incredibly well by giving the user full control over what should be evaluated. But Clojure's focus on immutability also makes this work really well. I rarely run into issues where I'm still depending on a var that's been removed and then there's still the reloaded workflow [2].

Overall I think notebooks are currently a great improvement for people that would otherwise create plain scripts – working on it is a lot quicker when you have an easy way to just execute parts of it. Plus there's the obvious benefit of interleaving prose and results. That doesn't mean we should not be thinking about addressing the hidden state problem but I think notebooks do add a lot of value nevertheless.

[1] https://clojure.org/guides/repl/introduction

[2] http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-r...

Post reply on HN