Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

github.com

Post reply on HN
Print view View original

31–40 of 71 posts

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#31

Post by crabbone »

gnulinux wrote: post #17
mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

It's a literate programming tool. If you find literate programming useful (such as Donald Knuth's Latex) then you can write a Jupyter notebook, add text, add latex, titles, paragraphs, explanations, stories and attach code too. Then, you can just run it. I know that this sounds pretty rare but this is mostly how I write code (not in Jupyter notebook, I use Markdown instead and write code in a combination of Obsidian…

I have to disagree... Literate programming is still programming: it produces programs (but with an extra effort of writing documentation up-front).

Jupyter is a tool to do some exploratory interactive programming. Most notebooks I've seen in my life (probably thousands at this point) are worthless as complete programs. They are more akin to shell sessions, which, for the most part, I wouldn't care storing for later.

Of course, Jupyter notebooks aren't the same as shell sessions, and there's value in being able to re-run a notebook, but they are so bad at being programs, that there's a probably a number N in low two-digits, where if you expect to have to run a notebook more than N times, you are better off writing an actual program instead.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#32

Post by big-chungus4 »

iamleppert wrote: post #28

Jupyter notebooks are missing strict types, a linter and unit tests. When can those features be added?

vscode jupyter uses the same extensions as vscode, so you can get a linter and scrict type checking. Not sure about tests though

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#33

Post by crabbone »

__MatrixMan__ wrote: post #10
mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

I think there are places where the figure-it-out-in-a-notebook part is one person's job, and then including it in a pipeline is another person's job. If they can call the notebook like a function, the second person's job becomes much easier.

I've been that person, and no it doesn't. It makes my life suck, if I have to include a notebook instead of an actual program in a larger program. Notebooks don't compose well, they are too dependent on the specifics of the environment in which they were launched, they have excessive source code that's also machine-generated and is hard to work with for humans.

As a stop-gap solution, for cases like a single presentation / proof-of-concept that doesn't need to live on and be reused -- it would work. Anything that doesn't match this description will accumulate technical debt very quickly.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#34

Post by abdullahkhalids »

crabbone wrote: post #31
gnulinux wrote: post #17

Earlier quoted context omitted.

It's a literate programming tool. If you find literate programming useful (such as Donald Knuth's Latex) then you can write a Jupyter notebook, add text, add latex, titles, paragraphs, explanations, stories and attach code too. Then, you can just run it. I know that this sounds pretty rare but this is mostly how I write code (not in Jupyter notebook, I use Markdown instead and write code in a combination of Obsidian…

I have to disagree... Literate programming is still programming: it produces programs (but with an extra effort of writing documentation up-front). Jupyter is a tool to do some exploratory interactive programming. Most notebooks I've seen in my life (probably thousands at this point) are worthless as complete programs. They are more akin to shell sessions, which, for the most part, I wouldn't care storing for later.…

> Don’t get discouraged because there’s a lot of mechanical work to writing. There is, and you can’t get out of it. I rewrote A Farewell to Arms at least fifty times. You’ve got to work it over. The first draft of anything is shit. Ernest Hemingway

This is how all intellectual work proceeds. Most of the stuff you write is crap. After many iterations you produce one that is good enough for others. Should we take away the typewriter from the novel writers too, along with Jupyter notebooks from scientists, because most typed pages are crap?

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#35

Post by kremi »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

Some of the replies here are pretty good, I basically agree with “if it works for your data scientists then why not”.

I’m actually a software developer with 10 years experience and also happen to do data science. And found myself in situations where I parametrized a notebook to run in production. So it’s not that I can’t turn it to plain python. The main reasons are

1. I prototype in a notebook. Translating to python code requires extra work. In this case there’s no extra dev involved, it’s just me. Still it’s extra work.

2. You can isolate the code out of the notebook and in theory you’ve just turned your notebook into plain py. You could even log every cell output to your standard logging system. But you loose context of every log. Some cells might output graphs. The notebook just gives you a fast and complete picture that might be tedious to put together otherwise.

3. The saved notebook also acts as versioning. In DS work you could end up with lots of parameters or small variations of the same thing. In the end what has little variations I put in plain python code. What’s more experimental and subject to change I put in the notebook. In certain cases it’s easier than going through commit logs.

4. I’ve never done this but a notebook is just json so in theory you could further process the output with prestodb or similar.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#36

Post by Kalanos »

ziddoap wrote: post #25
Kalanos wrote: post #24

Earlier quoted context omitted.

who is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.

What does any of this have to do with disclosure?

calling attention to disclosure suggests bias. i'm obviously saying that i trust him not to be biased.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#37

Post by Kalanos »

reeboo wrote: post #30
Kalanos wrote: post #23

Is this still being developed? The last commit to the main library was 5 months ago and its tied to exceptions/tests.

It's a thin wrapper around notebooks. Does it really need more features? Not saying that it couldn't, but it is feature complete for what its job is.

things break due to shifting dependencies.

also, if it isn't maintained by the company that made it, then it is a good sign that they are no longer using it. it suggests that there is a better solution elsewhere.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#38

Post by ogrisel »

iamleppert wrote: post #28

Jupyter notebooks are missing strict types, a linter and unit tests. When can those features be added?

With papermill you can parametrize a notebook and run it on different inputs to check that it is not raising uncaught exceptions. This can be wrapped to be part of a pytest test suite, possibly via a some ad-hoc pytest fixture or plugin.

If the notebooks themselves contain assertions to check that expectations on the outputs are met, then you have an automated way to check that the notebooks behave the way you want on some test inputs. For long notebooks, this is more like integration/functional tests rather than unit tests, but I think this is already an improvement over manually run notebooks.

Note sure about strict types: you mean running mypy on a notebook? Maybe this can be helpful:

- https://pypi.org/project/nb-mypy/

About linters, you can install `jupyterlab-lsp` and `python-lsp-ruff` together for instance.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#39

Post by __MatrixMan__ »

crabbone wrote: post #33
__MatrixMan__ wrote: post #10

Earlier quoted context omitted.

I think there are places where the figure-it-out-in-a-notebook part is one person's job, and then including it in a pipeline is another person's job. If they can call the notebook like a function, the second person's job becomes much easier.

I've been that person, and no it doesn't. It makes my life suck, if I have to include a notebook instead of an actual program in a larger program. Notebooks don't compose well, they are too dependent on the specifics of the environment in which they were launched, they have excessive source code that's also machine-generated and is hard to work with for humans. As a stop-gap solution, for cases like a single presenta…

I sort of suspected that adding parameters was not the end of the story. My experience with this was just "make it work with papermill", so the notebooks I tested with were nice and self contained.

Although it does seem like packaging dependencies and handling parameters are separate problems, so I'm not sure if papermill is to be blamed for the fact that most notebooks are not ready to be handled like a black box, even after they're parameter-ready. Something like jupyenv is needed also.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#40

Post by jamesblonde »

reeboo wrote: post #29
mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

As an MLE who comes from backend web dev, I have flip-flopped on notebooks. I initially felt that everything should be in a python script. But I see the utility in notebooks now. For notebooks in an ML pipeline, I find that data issues are usually where things fail. Being able to run code "up to" a certain cell and create plots is invaluable. Creating reports by creating a data frame and displaying it as a cell is al…

I agree completely with this. Papermill output is a notebook - that is the log file. You can double click on it, it opens in 1-2 seconds and you can see visually how far your notebook progressed and any plots you added for debugging.
Post reply on HN