Live data from Hacker News

The Future of Notebooks: Lessons from JupyterCon

willcrichton.net

111–120 of 158 posts

Re: The Future of Notebooks: Lessons from JupyterCon

#111
As the speaker for the scheduling notebook talk that got referenced here, I can probably give a few insights about how we're using notebooks. A lot of the risks and concerns brought up here were talked about in the session. The slides ended up on https://conferences.oreilly.com/jupyter/jup-ny/public/schedu... (and hopefully the talks themselves will get posted soon).

In particular I referenced how we treat and emphasize notebooks as an integration tool which acts as a good place to combine actions with documentation, visuals, and output logs. There's a section on Integrating Notebook which outlines how we approach this. There's also a strong emphasis on pushing complexity and shared code into the repositories housing notebooks. Effectively you end up with a lot of same best-practices found in non-notebook development, and the same abuses that lead to unmaintainable code -- which sometimes is needed in the short term.

So far we've had a lot of success with notebooks in production as parameterizable templates, or as a way to easily produce scheduled reports or machine learning experiments. Many users just provide the parameters while supporting teams provide the tested templates. Other users like being able to simply schedule their iterated work without needing to translate to another medium. One of the biggest wins though is gaining a shared interface for debugging, experimenting, and reusing code by having notebooks as the output artifacts of execution (even if it's just executing some other code elsewhere on behalf of the user). Papermill made a lot of this possible by separating input notebooks from output notebooks, and by being able to inject runtime values into those output notebooks.

Re: The Future of Notebooks: Lessons from JupyterCon

#113
post #52

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…

Jupyter notebooks are for data science, mostly because visualization is required. Something you do once, report it and it is done. Itsn't make sense to use Jupyter for other stuff. It doesn't make much sense to use them for training big models of deep learning because there are better tools for that.

> Something you do once, report it and it is done.

I have been having a hard time getting comfortable with that idea. There is usually "a lot of" untested and unreviewed code in the notebook that produces the analysis.

Re: The Future of Notebooks: Lessons from JupyterCon

#114
post #111

As the speaker for the scheduling notebook talk that got referenced here, I can probably give a few insights about how we're using notebooks. A lot of the risks and concerns brought up here were talked about in the session. The slides ended up on https://conferences.oreilly.com/jupyter/jup-ny/public/schedu... (and hopefully the talks themselves will get posted soon). In particular I referenced how we treat and emphas…

We also are doing a blog series on how we're approaching notebooks at https://medium.com/netflix-techblog/notebook-innovation-591e... (the scheduling post is the second in the series). Also I'd recommend listening to talks from the conference with an open mind. There were a lot of great discussions and positive energy around notebooks.

Hopefully these help describe a few of the new patterns and tools available in the notebook space.

Re: The Future of Notebooks: Lessons from JupyterCon

#115

Earlier quoted context omitted.

Jupyter is also really useful for figuring out how you want to do something before you copy it to your IDE. Every time I work with a new API or library I try it out in jupyter first.

What are the benefits of this vs just an integrated repl?

For me it is having a complete view of what I have done previously and rerunning past codeblocks by making changes.

Re: The Future of Notebooks: Lessons from JupyterCon

#116

Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.

I also work in scientific computing, and I love Jupyter. I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work. This potentially affects reproducibility, which is a tenet of…

> In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand.

Laughing, politely, at that summary of "history." Software engineering started when there were no screens. Code was written on typewriters (called "terminals") and then printed to cards or long rolls of paper (and occasionally using switches to indicate the 1's and 0's you wanted set as bits in your code). Before the Mac a "screen" of code was pretty universally 24 lines of green or white text on a black background, 80 monospace characters per line. A few specialized terminals had 25x80 so they could show a status line below the 24x80 standard area, but they were rare beasts. Yes, there were a few systems like Plato that predated the Mac but if you need a tl;dr it's the observation that, like most revisionist histories, this one is unfortunately total nonsense when looked at in the context of actual history (even if lots of people who didn't experience the actual history believe it).

Re: The Future of Notebooks: Lessons from JupyterCon

#117
post #94
post #92

Earlier quoted context omitted.

Jupyter code is draft code. Draft code rarely makes into clean code, because it requires extra tedious effort. Most of us are too overworked to have the time or energy to perform it. Alternatively one can write exploratory code as unit tests and run them inside an IDE. When the exploration is done, three quarters of the tedious cleanup effort is already done. There are already unit tests. There are already APIs exerc…

Vscode has decent Jupyter support with its python plugin. What I really missed my jupyter notebooks was a world class editor with keyboard shortcuts and code completion. So I went the other route, running jupyter notebooks in vscode. I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa. https://github.com/nojvek/vscode-ipynb-py…

Wow! This is insanely cool! You should package this up as a VS Code plugin so more people can use it. At a minimum, please post a license.txt file in your repo so folks can actually use it (MIT or similar is pretty easy and common for something like this, without a license.txt file it's not clear whether others can actually use it and many places that means they can't).

Re: The Future of Notebooks: Lessons from JupyterCon

#118

Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.

This is a really important comment. I started my career in scientific computing, went over to software/web development for a while, and am back over to scientific computing. I've been involved in teaching efforts lately, mainly for analysts who want to do data wrangling and stats, and I think it is a good environment for some of what they want to do.

I never used Jupyter before getting involved in this kind of coding, data wangling and stats. And it took me a while to get used to it. The "lack of state" is confusing, as well as the apparent non-linearity of a jupyter notebook (it can appear that certain commands are done in order from top to bottom, when if you look at the number next to them, they may have been executed in a completely different order). I actually chalked this up to my background in software development, figuring I was more wired to think of state as something that would be cleared every time a script was run. Analysts (statisticians, engineers, data scientist types) like to interact back and forth very iteratively with their data, and I think they're less likely to be tripped up by some back of mind assumption that you can determine the state of variables and so forth by looking at a bunch of commands as if they were run top to bottom in a file.

It got me thinking about when I'd use jupyter notebooks for my own work. In short, if I were writing a program, even just a one page script, I wouldn't use an interactive notebook (I have opened .py files in jupyter and just used it as a text editor).

But if I wanted to "do my math homework"? A one off where I need to get an answer to some complicated questions that will require a lot of data wrangling and stats? Yeah, I'd probably reach for jupyter notebook or something like it.

Otherwise? I can't remember the O'Reilly book where this was written (sorry I wish I had the cite) but the author wrote that his favorite dev environment is python and a text editor. That's still the case for me as well.

I hate to conclude with the disappointing "the problems are when you use it for the wrong task", mainly because I've objected when people have made this argument in the past. If a tool tends to get used for the wrong task, that's partly a problem with who is using it... but we shouldn't let the tool itself off the hook too easily. It's still worth thinking through. I'm worried I'm falling into the trap of making this argument when I like the tool, and objecting to it when I don't. I'll have to think about that a bit.

As it stands, though, I haven't had a huge problem with Jupyter Notebook, and I like it a lot, probably because I don't use it when Python + vi (or a "better" but still basic text editor) would do the trick.

Re: The Future of Notebooks: Lessons from JupyterCon

#119
As a pretty heavy notebook user (for computer vision and machine learning), I have to say that all of these new features are nice but they don't address the real problems with the current ecosystem. Namely, there's no good way to transition exploratory notebook code to python modules. We need linting and refactoring tools in the notebook and something better than `aimport` for moving code into separate `.py` files.

Re: The Future of Notebooks: Lessons from JupyterCon

#120

Earlier quoted context omitted.

What? What are you using Jupyter notebooks for where you want maintenance? They should be records of data analysis/ procedures, not code that runs in production or something.

Maybe read the article. They are experimenting with putting the notebooks directly into production ala bash script. I don’t think this is a great idea either.

Yeah, I read it after, silly of me to comment first.
Post reply on HN