JupyterCon: I don't like Notebooks [slides]

Fascinating presentation! Joel is such a meme factory :-) Has anyone done a "rejected memes" section at the end of a technical talk before?

On topic: I applaud the effort to set things straight, but I'm afraid that ship has sailed. Hacky "Kaggle notebook solutions" are now Data Science. That's what the term evolved to mean, and all the rest of the impedance highlighted in this presentation follows.

If your work involves creating well-designed, well-factored, tested, reusable ML software that is meant to be integrated, picked apart, extended and applied over time in practice (as opposed to be submitted once to an oracle in order to claim "SOTA"), you better come up with another term. And obviously Notebooks are not a good fit, beyond as a tool for documentation & reporting (which Joel correctly calls out as genuinely useful).

See a previous post of mine (https://news.ycombinator.com/item?id=17840216) for my general (frustrating) experience with notebooks.

I'll add positives notes, though:

-They're excellent for beginner tutorials that explain an abstract concept or a library's use cases

-They're very well suited for academic peer-review. It's basically a way to say: here's what we obtained, here are the exact steps we did to obtain it, you can obtain it as well if you do exactly as we did. You'd think this reproducibility requirement would be common sense but in practice you're usually thankful if the data is available, or the software is released and works.

Ironic this pops up today when a colleague has this scenario happen...

BA: We found this duplicate field error in our database, do you know about this.

DS: I told you about this 2 years ago.

BA: Oh so you have a script to produce all the duplicate records?

DS: No I did this 2 years ago I don't know what version of Python or any of the libraries I used for the notebook.

In my experience a lot of data scientists like this work alone on business problems - if they were in a more collaborative environment they'd probably run into the problems that Joel is discussing a lot more and be more open to build libraries.

Hmm, agreed. But before you share a notebook, restart kernel and rerun all cells, fix all errors, and then share... Solves many of his gripes.

I use notebooks all the time, but I put repetitive code in a module, try do put all functions at the start etc. It is much easier to share data science experiments, but indeed notebooks can be used and presented in horrible ways.

So... now we write a presentation on how we can horribly abuse VSCode + an iPython console and call it even?

I love notebooks, but I agree with you. I like to use notebooks to instrument and debug modular models. Top down design, which is what I was taught years ago when I was learning C/C++ seems harmful when paired with out of order execution.