Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

github.com

Post reply on HN
Print view View original

11–20 of 71 posts

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#11

Post by jdiez17 »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

There are a lot of people who are not expert Python programmers, but know enough to pull data from various sources and make plots. Jupyter{Notebook,Lab} is great for that.

As you say, from a programmer's point of view the logical thing to do is to convert the notebook to a Python module. But that's an extra step that may not be necessary in some cases.

FWIW I used papermill in my Master's thesis to analyze a whole bunch of calibration data from IMUs. This gave me a nicely readable document with the test report, conclusions etc. for each device pretty easily.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#12

Post by morkalork »

I once built an unholy combination of papermill and nbconvert to mass produce monthly reports using a "template" notebook. All the code was imported from a .py file so the template just took a client ID as input and called out to render_xyz(...) in each section. It was nice because it produced a bunch of self-contained static files and wrote them to a network drive. It was definitely a solution to the problem.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#13

Post by p4ul »

notpushkin wrote: post #7

> Do you want to run a notebook and depending on its results, choose a particular notebook to run next? Hell no. I want to rewrite all that as a proper script or Python module.

Indeed! I feel like we as a community have taken a wrong turn with our use of notebooks. I think they have benefits in some specific use cases (e.g., teaching, demos, etc.), but otherwise, I think they mostly encourage bad practices for software development.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#15

Post by crystal_revenge »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

I agree. I was at a company where some DS was really excited about Papermill, and I was trying to explain that this is an excellent time to stop working in a notebook and start writing reusable code.

I was aghast to learn that this person had never written non-notebook based code.

Code notebooks are great as notebooks, but should in no way replace libraries and well structured Python projects. Papermill to me is a huge anti-pattern and a sign that your team is using notebooks wrong.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#16

Post by mooreds »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

It's the same tradeoff of turning an excel spreadsheet into a proper program.

If you do so, you gain:

* the rigor of the SDLC

* reusability by other developers

* more flexible deployment

But you lose the ability for a non-programmer to make significant changes. Every change needs to go through the programmer now.

That is fine if the code is worth it, but not every bit of code is.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#17

Post by gnulinux »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

It's a literate programming tool. If you find literate programming useful (such as Donald Knuth's Latex) then you can write a Jupyter notebook, add text, add latex, titles, paragraphs, explanations, stories and attach code too. Then, you can just run it. I know that this sounds pretty rare but this is mostly how I write code (not in Jupyter notebook, I use Markdown instead and write code in a combination of Obsidian and Emacs). To me, code is just writing, there is no difference between prose, poetry, musical notation, or computer programming. They're just different languages that mean something to human beings and I think they're done best when they're treated like writing.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#18

Post by swalsh »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

My experience is more with Databricks, and their workflow system... but the concept is exactly the same.

It let's data scientists work in the environment they work best in, and it makes it easier to productionize work. If you seperate them, then there's a translation process to move the code into whatever the production format is which means extra testing, and extra development.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#19

Post by z3c0 »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

Parameterizing notebooks is a feature common to modern data platforms, and most of its usefulness comes from saving the output. That makes it easier to debug ML pipelines and such, cos the code, documentation, and last output are all in one place. However I don't see any mention of what happens to the outputs with this tool.
Post reply on HN