Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
1–10 of 71 posts
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#2Papermill is great but has quite some limitations because it spins up a new process to run the notebook:
- You cannot extract live variables (needed for testing)
- Cannot use pdb for debugging
- Cannot profile memory usage
You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine).
Disclaimer: I'm the author of this package
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#3Papermill is great but has quite some limitations because it spins up a new process to run the notebook: - You cannot extract live variables (needed for testing) - Cannot use pdb for debugging - Cannot profile memory usage You can do all of that with ploomber-engine ( https://github.com/ploomber/ploomber-engine ). Disclaimer: I'm the author of this package
Not disclosed in this comment is that edublancas is
>Ploomber (YC W22) co-founder.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#4Papermill is great but has quite some limitations because it spins up a new process to run the notebook: - You cannot extract live variables (needed for testing) - Cannot use pdb for debugging - Cannot profile memory usage You can do all of that with ploomber-engine ( https://github.com/ploomber/ploomber-engine ). Disclaimer: I'm the author of this package
iirc, a few years back I was able to do all of these things with the Papermill IPython runtime.
Papermill is great, but yes: lots of room to hack on it and make it better.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#5Papermill is great but has quite some limitations because it spins up a new process to run the notebook: - You cannot extract live variables (needed for testing) - Cannot use pdb for debugging - Cannot profile memory usage You can do all of that with ploomber-engine ( https://github.com/ploomber/ploomber-engine ). Disclaimer: I'm the author of this package
iirc, a few years back I was able to do all of these things with the Papermill IPython runtime. Papermill is great, but yes: lots of room to hack on it and make it better.
has papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#6What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#7> Do you want to run a notebook and depending on its results, choose a particular notebook to run next?
Hell no. I want to rewrite all that as a proper script or Python module.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#8What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
It might be a pretty useful tool for education. College courses related to Python and AI on Coursera have heavily used Jupyter Notebook for assignments and labs.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#9What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
I used papermill a while ago to automate a long-running python-based data aggregation task.
Airflow would log in remotely to the server, kick-off papermill and track it's progress. Initially I wanted to use pure python, but the connection disconnected frequently disallowing me to track the progress, and also jupyter enabled quick debugging where something went wrong.
Not one of my proudest moments, but it got the job done.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#10What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
I think there are places where the figure-it-out-in-a-notebook part is one person's job, and then including it in a pipeline is another person's job.
If they can call the notebook like a function, the second person's job becomes much easier.