JupyterCon: I don't like Notebooks [slides]

I agree, many (most?) data scientists spend most of their time coding, it's ridiculous how little time they're willing to hone this skill.

Data science practices are similarly inexact - a lot of good decision making comes from experience, knowing when to apply each tool to a specific problem, when to just throw in a hack etc.

I don't even like it for initial experimentation and exploration. Much prefer the command line repl (the jupyter/ipython one). Faster feedback and easier navigation then you can just export the history when you're ready to capture and make permanent some workflow. It still requires going through and picking out the important bits, but reading the history like a story of commands is generally enough to pick out the important parts.

The only thing I would use notebooks for is demonstration / teaching.

This was great, bravo! While I am not necessarily the audience intended. I've been following some Notebook development. I have always thought of it as convoluted and pedantic. I don't mean to distract from some very solid and well intentioned work. Just that it added a layer of complexity and turned out not to be as portable as advertised.

Notebooks may be suitable for scratching things, but not much else. The problems noted by the OP are very serious, specially in the scientific world, where many do not have proper software engineering skills. The real issue is that newcomers do not know better, and so they don't realize the damage they're inflicting upon themselves and others before it's too late (irreproducible code, hidden states, dep. management, etc).

Even the fast.ai library, which is a wonder, has broken notebooks. For those who try to follow the course at home, trying to run the notebooks is frustrating, as things are out of order and so errors pop up all the time. Jeremy is a wonderful teacher, but compare following a Fast.ai course video, which uses notebooks, to following a python video from, e.g., Raymond Hettinger [2], which uses sphinx and a shell. While the documentation style and ugly shell don't look nearly as cool, they are so much clearer and better structured.

Notebooks become popular because they fill one gap that was left uncovered. As the scientific community moves away from Matlab into Python and R, reading code - pushed, amongst others, by the popularity of Github - becomes a day-to-day activity. Matlab scripts were easily explorable because users would load them, set breakpoints here and there, and look at results interactively - exactly what notebooks aim to provide. The difference is that what used to be breakpoints now become cells, comments now turn into Markdown and figures are inlined to add an extra layer of convenience. Yet all the awful problems of sloppy Matlab development are now masked, marketed as something fancy and start to pollute the Python dev. environment. Reproducibility and testing are gone, dependency management (which is not required for Matlab) breaks down completely, documentation is non-existent, and sharing becomes heavily constrained.

Notebooks may be suitable for scratching things, but not much else, at least nothing serious. Hopefully the slideshow above gets the attention it deserves. And in all cases, kudos for the Jupyter dev team for fighting the good fight, even with the drawbacks of their experiment.

[1] http://course.fast.ai/lessons/lesson3.html [2] https://www.youtube.com/watch?v=9zinZmE3Ogk

RMarkdown have all the advantages of notebook and without most of its problems.

- code and document mixed. You have full markdown syntax, chunks, titles, table of contents. You can also convert RMarkdown into a script with comment if you need.

- To render a RMarkdown, it will run in a separate environment, start from scratch. It's assumed you should make it reproducible, this is good for report/sharing.

- I also use RMarkdown to write code, and run code/code chunks interactively in a session. I can write plan, notes, references, TODO in document, test code in chunks, execute code chunks in any order. Basically you just code and document. In the end you can turn it into a report, or refactor the tested code into functions and scripts. I kept the original RMarkdown as design document, which have all the original notes and previous version of code.

- It's plain text so version control is fully supported.

RMarkdown started to support python, but it may be preliminary for now.