Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What’s wrong with computational notebooks?

web.eecs.utk.edu

Post reply on HN
Print view View original

201–210 of 223 posts

Re: What’s wrong with computational notebooks?

#201

Post by BeetleB »

comment_guy wrote: post #171
vnpc1 wrote: post #110

Earlier quoted context omitted.

> I don't get why anyone one who knows how to use an IDE would ever use a notebook, The Python IDEs for data science are mostly garbage - if you have any recommendations, I'm all ears because I really don't like notebooks but still keep switching between jupyter and vscode depending on what I'm working on.

I use IntelliJ for all my work, data or normal dev stuff, and it works great (all is python). Maybe there is just a workflow issue here where people are used to saving their data as they go in cells. I just write my algorithms all the way through, get a subset of data to debug against, then use the debugger to help me see what mistakes I made. I always run my code all the way through and only stop at the step I'm deb…

> I use IntelliJ for all my work, data or normal dev stuff, and it works great (all is python).

Can it show me inline plots and allow me to embed rendered formulae written in LaTex, images and video in between lines?

This is the reason people like notebooks.

Re: What’s wrong with computational notebooks?

#202

Post by btbuildem »

rabryan35 wrote: post #7

No mention of https://observablehq.com notebooks? They’re the best I’ve found in the “Share and collaborate” and “As products” category. JupyterLab is still pretty great for exploratory stuff, but visualization possibilities in observable are incredible.

They're a walled garden and to be honest I kinda hate them.

Bostock had lots of accessible D3 examples before, now it's all on that "platform".. sure it's slick, but overall loss for the ecosystem IMO.

Re: What’s wrong with computational notebooks?

#203

Post by bobbylarrybobby »

The problem of notebooks has been solved by the Python extension in Visual Studio Code (and some other editors too, although VS Code is the one I'm most familiar with).

Editing an ordinary Python file, if you insert the comment "# %%", you turn everything between that comment and the next "# %%" (or the end of the file) into a code cell that can be submitted to the ipython kernel, just as in a Jupyter notebook. The editor splits into two halves, the left half your Python file and the right half the Jupyter notebook window with submitted code and formatted output (e.g., DataFrames look pretty, plots display normally, etc.). When you're done running everything, you can export the result as a Jupyter notebook. Because you're editing an ordinary Python file, standard features like version control and importing the file you're editing into other files (you cannot normally import .ipynb files IIRC) work normally.

And of course since VS Code is a real editor/IDE, you can double click a file and have it open right up (no resorting to a Terminal to start your Jupyter session) and you get syntax themes, a built in Terminal, a git UI, code snippets, documentation on hover, vim mode if that's your thing, etc.

The only downside I've found is that the Python extension doesn't incorporate ipython's autocomplete in its own autocompletion, but that's a small price to pay for getting to treat .py files as notebooks.

Re: What’s wrong with computational notebooks?

#204

Post by etangent »

Sean1708 wrote: post #135
etangent wrote: post #100

Earlier quoted context omitted.

I keep seeing the complaint about state and I am getting sick of it. JUST HIT "RESTART KERNEL AND RERUN ALL CELLS," HOW HARD CAN IT BE?

Maybe I'm missing a joke here, but if that's your workflow then there's absolutely no advantage to notebooks over something like Spyder or even VS Code.

No, that's not the workflow. You work in the notebook as normal but from time to time (say every two hours) rerun the whole thing.

One advantage of this is that it forces you to name your variables such that they don't overwrite each other. Further down the line this enables sophisticated comparisons of states (e.g. dataframes) before and after (something data scientists need)

Re: What’s wrong with computational notebooks?

#205

Post by etangent »

gdy wrote: post #107
etangent wrote: post #100

Earlier quoted context omitted.

I keep seeing the complaint about state and I am getting sick of it. JUST HIT "RESTART KERNEL AND RERUN ALL CELLS," HOW HARD CAN IT BE?

If you have a few long data loading and preprocessing steps it's a pain to wait for them to run again, people try to avoid it. When something odd begins to happen, they don't immediately consider the possibility that it's not their bug and waste time trying to 'debug' the problem instead of just rerunning the notebook.

You don't reload every time you write a line of code. Nobody's insane like that. You reload every two hours or so. This is good enough for most except most extreme data sets.

Re: What’s wrong with computational notebooks?

#206

Post by etangent »

dkersten wrote: post #141
etangent wrote: post #100

Earlier quoted context omitted.

I keep seeing the complaint about state and I am getting sick of it. JUST HIT "RESTART KERNEL AND RERUN ALL CELLS," HOW HARD CAN IT BE?

Why are you even using notebooks at all then?

> One advantage of this is that it forces you to name your variables such that they don't overwrite each other. Further down the line this enables sophisticated comparisons of states (e.g. dataframes) before and after (something data scientists need)

Also, not sure about you, but I like seeing all of my outputs on a single browser page without having to write any glue code whatsoever.

Re: What’s wrong with computational notebooks?

#207

Post by thenipper »

pantsforbirds wrote: post #198
snapetom wrote: post #42

Earlier quoted context omitted.

I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…

My last job I spent 80+% of my time productionizing models and notebooks. It was an absolute nightmare. Everyone had slightly different preprocessing hacks for different stages and things were always working fine locally, but I couldn't replicate the results in docker containers. I am very happy to be out of that business.

are you me?

Re: What’s wrong with computational notebooks?

#208

Post by fock »

archi42 wrote: post #179
PeterisP wrote: post #154

Earlier quoted context omitted.

The key factor is iteration speed. If step A takes 5 minutes (and 5 minutes is a very short time) and I want to experiment on step B, then I don't want to rerun step A each time while I'm writing and running code that helps me understand what step B is going to be; I'd want that to be interactive and immediate, not have each rerun take 5 minutes. Storing/loading to disk is not a good option because all the data that…

Ah, I think I have a hugely different approach to data processing: For my work I often have a very good idea what the output should look like, and what transformations are required on the input to get there. E.g. when processing log files to generate an overview page, or (as I'm doing right now) adding a target to binutils (assembler, linker,...). (Obviously I'm not a data scientist ;-) With what you describe, intuit…

https://joblib.readthedocs.io/en/latest/ (though it has a large overhead)

Re: What’s wrong with computational notebooks?

#209

Post by bart_spoon »

I'm not sure I understand the issue about the user repeatedly tweaking parameters for their data visualization. If anything, that is a reason notebooks are so nice. The repeated tweaks are due to the notebook format, its because that's an inherent part of the data visualization process, where the end result of a particular parameter choice is hard to predict how it will look with a given data set. So the same process would occur whether one was using a notebook or a script, but with a script it becomes much more cumbersome to actually see the result. In a notebook, the parameter tweaking for a data visualization is immediately followed by the result.

I definitely agree with most of the other points though.

Re: What’s wrong with computational notebooks?

#210

Post by ktpsns »

nl wrote: post #20
ktpsns wrote: post #11

Obviously it's pretty hard to make general criticism of the Notebook GUI. This is especially without comparing to a specific other user interface for data scientists, such as a traditional REPL terminal, or some other command line tools? The Python world gives a good example about the sheer complexity of a notebook infrastructure. The is IPython, there is Jupyter Notebook, JupyterLab. There is even stuff like the Sag…

Jupyter Notebook/Jupyter Lab has replaced IPython as the notebook front end. I suspect 90%+ of Python Notebook work is done in Jupyter/Jupyter Lab (or things built on it like Google Collab/Kaggle Kernels). traditional notebook GUIs of Matlab and Mathematica, something we used in the 90s and 2000s. From my feeling, they were more robust, could handle large data better I've done 10s of terabyte analysis on Jupyter (Spa…

You probably were careful enough to understand the limits of the Jupyter server and client (frontend).

It's easy to screw up a terminal application in data science when dumping a large array. Many REPLs cannot handle this properly (and CTRL+C won't work). It's easy to test this: What does your favourite notebook do when you call some command such as (pseudocode/python here)

    print(range(int(1e7))) # or 1e8
In this particular example, the python CLI seems to handle keyboard interrupts fine when the terminal (or RAM) is flooded.
Post reply on HN