Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What’s wrong with computational notebooks?

web.eecs.utk.edu

Post reply on HN
Print view View original

51–60 of 223 posts

Re: What’s wrong with computational notebooks?

#51

Post by bryanhpchiang »

I interned @ Google AI last summer; used notebooks nearly everyday. Estimated productivity gain is 3-5x.

Biggest tip I have is to turn auto reload on, then write the bulk of your code as modular functions and call functions within your notebooks. Keeps the notebook tidy and it’s easier to push your code this way.

It’s also easier for sharing since most people viewing your notebooks (mentors, people outside your team) are interested in results/artifacts such as metrics, generated text, images, audio, which notebooks display well (not your code).

Re: What’s wrong with computational notebooks?

#52

Post by sillysaurusx »

tgb wrote: post #47
sillysaurusx wrote: post #37

Earlier quoted context omitted.

It's best to think of the notebook as a REPL. So you'd want to run `del foo` on the old name. In fact, this is a good counterexample. Why should the notebook delete the old variable name? What if its value is a thread currently executing? Notebooks are REPLs, and it's better to get used to that than to try to enforce some confusing time traveling.

But, strangely, Jupyter doesn't also give you a REPL (like, say, R Studio does). I'm always making new cells in the middle to output the column names of my spreadsheet, and then I have to delete them. I used to just always have an ipython REPL running and test things out in there as I write. You can start a ipython instance on the same kernel but I found that messed up my plots when I did that IIRC.

Yes! I've wanted this too.

Colab has a nice feature that's close to this: Insert -> Scratch code cell

Re: What’s wrong with computational notebooks?

#54

Post by breatheoften »

The reality is — notebooks are and need to be developed as an app platform ...

In order to do notebooks properly — you need:

1. discovery (Ideally static discovery) of all the state the notebook needs, and the bulk of state the notebook will/could manipulate during its execution. Your container needs to intercept the filesystem and the networking apis that will be invoked so that a determination of the state that results from these operations can be observed by the runtime and shimmed appropriately for reproducibility and for performance optimization

2. The notebook (and the runtime inferred model of all the required inputs) needs to be repo stable — I Should be able to write a notebook app that reads from the file system on my development host, deploy it somewhere, and the runtime should take care that wherever that however that post deployment file system read is implemented matches my local development semantics

3. Pplatform level dependency graph needs to exist to model re-execution requirements automatically — incorporating code changes and external state

Apple could build this And “notebook-os” would be the correct conceptual framework for it ... anything less is always going to leave us severely wanting

Re: What’s wrong with computational notebooks?

#56

Post by andrew_n »

I used Mathematica’s notebook interface quite heavily 15-20 years ago; Jupyter’s interface is a clone of that in many ways.

At the time, my workflow was to use two different notebooks for everything: foo.nb and foo-scratch.nb. I’d get things working a piece at a time in foo-scratch.nb, not caring at all how it looked, not having to worry about leaving extra output or dead ends of explorations lying around; then the refined cells would be copied over to foo.nb, which would get pristine presentation, and which I could run top-to-bottom.

This workflow worked pretty well for me: very clean reproducible output, with the ability to easily refer back to all the steps of how I’d derived something, along with copious detailed private notes.

I never had to use it but I’m pretty sure each cell even had its modification time stored in the metadata in case I wanted to view a chronological history.

Re: What’s wrong with computational notebooks?

#57

Post by bb88 »

wwarner wrote: post #44

I have been heads down in jupyter for the past couple of weeks and I finally realized I just DO NOT LIKE IT AT ALL ! Cracks started appearing and then suddenly there was an avalanche of disappointment. The first crack -- it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. I imported all the TeX goodness, and it looked pretty nice, but I couldn't sh…

So once your code gets large enough that it doesn't fit neatly within a jupyter notebook, it's time to split the code out into another package, and then import it into your notebook.

The benefit here is that now your code and be used inside the jupyter notebook, and also inside a webserver say.

Re: What’s wrong with computational notebooks?

#58

Post by amiga_500 »

wwarner wrote: post #44

I have been heads down in jupyter for the past couple of weeks and I finally realized I just DO NOT LIKE IT AT ALL ! Cracks started appearing and then suddenly there was an avalanche of disappointment. The first crack -- it's almost impossible to build a nice presentation in Jupyter, because you always have to show your code and its stderr. I imported all the TeX goodness, and it looked pretty nice, but I couldn't sh…

I've started to use jupyter with kdb on the back end for analysis. For me I have some hope it will hit the sweet spot because:

1. kdb is "too obtuse" for many and python glue makes it more amenable

2. I can still have kdb functions in source code and call them from jupyter with pyq

3. I can do most of my "editing" in emacs to the kdb back-end, write python "libs" for parsing results, and just use jupyter as a fairly thin presentation layer

4. I can share notebooks with analysts who run the same jupyter server virtual env, so finally we can share notes

Will this add more value than just using kdb? Time will tell, hard to know right now.

I agree that the "ide" experience absolutely sucks.

Re: What’s wrong with computational notebooks?

#59

Post by comment_guy »

I don't get why anyone one who knows how to use an IDE would ever use a notebook, the coding experience is garbage in comparison. I understand they started as a way to get STEM kids coding quick, but now they are like a standard in data analysis and data science, with those people needing experienced devs to translate the notebook into production code. This just drives the silo walls up higher.

Re: What’s wrong with computational notebooks?

#60

Post by bb88 »

moultano wrote: post #5

I want a notebook where causality can only flow forward through the cells. I hate notebook time-loops where a variable from a deleted cell can still be in scope. 1. Checkpoint the interpreter state after every cell execution. 2. If I edit a cell, roll back to the previous checkpoint and let execution follow from there. I can't tell you how many times I've seen accidental persistence of dead state waste hours of peopl…

Makefiles have this issue too, sometimes things have been incorrectly made, and the dependencies in the makefile are wrong.

Unless it takes more than a few seconds to run a notebook, rerun every cell up to the point you're editing, always.

And then if it does take minutes, and you find yourself in an unexplainable rut, then run the entire notebook, and get a cup of coffee.

Post reply on HN