Notebooks Are McDonalds of Code

Notebooks are primarily a _teaching_ tool, not code or an experimentation interface. I know that's not how they're used, but it's really bizarre to me that people use them for exploratory data analysis. They're great for teaching and for _documenting_ the results of exploration, but fairly bad for any type of interactive data exploration, let alone actually writing code.

Frankly, notebooks are pretty terrible for exploratory data analysis, especially in python. The main issue is that they absolutely kill interactivity. Straight up CLI based ipython is far more interactive, and often better for exploratory data analysis. In a lot of ways, pretty much any other interface is better than code blocks that aren't independent but can be run independently.

Part of it is the interface (i.e. non-linear state, code blocks, etc), which is great for teaching, but bad for exploration. However, i.m.o. even more of it is the fact that it's browser based. E.g. folks often think it's best to use matplotlib in a notebook. However, using a notebook means you get very little interactivity out of a highly interactive library and remarkably poor performance out of something that's actually _very_ interactive and quite performant on standard Tk/Qt/etc backends.

If you ditch browser based stuff, you're usually far better off. This part is mostly just me griping, but I don't understand why we've stepped back 20 years in basic usability, interactivity, and performance by making everything use a web browser (and then making that not even remotely cross platform).

In one project, I had a new teammate ask me why almost our entire web backend is a single .js file mostly filled with SQL queries. He wanted to make it his personal project to refactor this into probably 20 different files, adding more layers between the handlers and the DB stuff, using a query builder, and also migrating to TS. When I asked him what's wrong with the current thing, he couldn't answer and gave up on this idea.

That backend had decent integration tests, and it took less than 5 minutes to add a small new feature. Most prod code doing comparable things where I work is worse-tested, less reliable, and at least 10X more expensive in terms of SWE-time, so I think modern programming standards are actually nonsense even though I can play along with them.

Notebooks are not IDE, so cannot be compared to ones. You can run notebook in vscode, and keep the best code practices. Experiment with code, incrementally move code chunks to external modules. Another great feature is ability to eaisly switch kernel to remote one without leaving IDE. Downside is that it can be really annoying to manipulate the code.

One of those complaints is due to an unfortunate implementation choice.

Out-of-date cells happen because Jupyter works like a buggy makefile that doesn't reliably rebuild dependencies, forcing you to run "make clean" when anything weird happens. There are better build systems.

Observable notebooks will automatically rerun cells that changed, like a spreadsheet. It works nicely for calculations that aren't too heavy, but it might not be what you want for a heavy batch job.

Their newer tool, Observable Framework, works more like a regular build system. You can still have it automatically build when you save a file in your editor.

A second complaint, that it's browser based, is basically an editor preference. You can open Jupyter notebooks in VS Code if you prefer.

I like this analogy a lot. They both give you a very intuitive feel for the data and calculations being applied.