For MLOps platforms, Papermill is the one of the reasons why we don't include experiment tracking out of the box any longer in Hopsworks. You can easily see the results of training runs as notebooks - including loss curves, etc. Any models that completed get registered in the model registry along with plots, a model card, and model evaluation/validation metrics.
Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
41–50 of 71 posts
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#42What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
I agree. I was at a company where some DS was really excited about Papermill, and I was trying to explain that this is an excellent time to stop working in a notebook and start writing reusable code. I was aghast to learn that this person had never written non-notebook based code. Code notebooks are great as notebooks , but should in no way replace libraries and well structured Python projects. Papermill to me is a h…
So you think it was a good move to scoff at someone for using a computer for their work in a way that is different from your preferences?
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#43What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
It's the same tradeoff of turning an excel spreadsheet into a proper program. If you do so, you gain: * the rigor of the SDLC * reusability by other developers * more flexible deployment But you lose the ability for a non-programmer to make significant changes. Every change needs to go through the programmer now. That is fine if the code is worth it, but not every bit of code is.
It also implies that an engineer has better understanding of what is supposed to be done and can discover all the error modes.
In my experience, most of the time the problem is in the input and interpretation of the data. Not fixable by a unit test.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#44I teach a lot using Jupyter. It is certainly possible to use SWE worst practices in Jupyter easily.
I am often in front of folks who "aren't computer programmers" but need to use Python tools to be successful. One of my covert goals is to teach SWE best practices inside of notebooks. It requires a little more typing but eases the use of notebooks, refactoring, testing, moving to scripts, and using tooling like Papermill.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#45I teach a lot using Jupyter. It is certainly possible to use SWE worst practices in Jupyter easily. I am often in front of folks who "aren't computer programmers" but need to use Python tools to be successful. One of my covert goals is to teach SWE best practices inside of notebooks. It requires a little more typing but eases the use of notebooks, refactoring, testing, moving to scripts, and using tooling like Paperm…
Have you considered using marimo notebooks?
https://github.com/marimo-team/marimo
marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale.
marimo notebooks are also readily parametrized with CLI arguments, so you can do: python notebook.py -- -foo 1 -bar 2 ...
Disclosure: I'm a marimo developer.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#46Earlier quoted context omitted.
I agree. I was at a company where some DS was really excited about Papermill, and I was trying to explain that this is an excellent time to stop working in a notebook and start writing reusable code. I was aghast to learn that this person had never written non-notebook based code. Code notebooks are great as notebooks , but should in no way replace libraries and well structured Python projects. Papermill to me is a h…
So you think it was a good move to scoff at someone for using a computer for their work in a way that is different from your preferences?
Notebooks are great as notebooks, but it's very well established, even in the DS community, that they are a terrible way to write maintainable, sharable, scalable code.
It's not about preference, it's objectively a terrible idea to build complex workflows with notebooks.
The "scoff" was in my head, the action that came out of my mouth was to help them understand how to create reusable Python modules to help them organize their code.
The answer is to help these teams build an understanding of how to properly translate their notebook work into re-useable packages. There is really no need for data scientists to follow terrible practices, and I've worked on plenty of teams that have successfully been able to onboard DS as functioning software engineers. You just need a process and a culture that notebooks cannot be the last stage of a project.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#47Earlier quoted context omitted.
It's a literate programming tool. If you find literate programming useful (such as Donald Knuth's Latex) then you can write a Jupyter notebook, add text, add latex, titles, paragraphs, explanations, stories and attach code too. Then, you can just run it. I know that this sounds pretty rare but this is mostly how I write code (not in Jupyter notebook, I use Markdown instead and write code in a combination of Obsidian…
I have to disagree... Literate programming is still programming: it produces programs (but with an extra effort of writing documentation up-front). Jupyter is a tool to do some exploratory interactive programming. Most notebooks I've seen in my life (probably thousands at this point) are worthless as complete programs. They are more akin to shell sessions, which, for the most part, I wouldn't care storing for later.…
Literate programming is not just "documentation + code" any more than a textbook you read about Calculus is "documentation + CalculusCode" or a novel is "documentation + plot". It goes way beyond that, using literate programming you can attach an arbitrary text that accompanies the code such that fragments of your code is simply one part of the whole text. Literate programming is not just commenting (or supercommenting), if it were, you could use comments, it's a practice of simply attaching fragments of code in a separate text such that you can then later utilize that separate text the same way you utilize code. When you write a literate program, your end goal is the text and the program, not just the program. You can write a literate program, and publish it as is as a textbook, poem, blog post, documentation, website, fiction, musical notation etc... Unless you think that all human writing is documentation then literate programming is not just documentation.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#48I teach a lot using Jupyter. It is certainly possible to use SWE worst practices in Jupyter easily. I am often in front of folks who "aren't computer programmers" but need to use Python tools to be successful. One of my covert goals is to teach SWE best practices inside of notebooks. It requires a little more typing but eases the use of notebooks, refactoring, testing, moving to scripts, and using tooling like Paperm…
Have you considered using marimo notebooks? https://github.com/marimo-team/marimo marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale. marimo notebooks…
thanks mate, exactly what I've been looking for
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#49In the past, I've used this to generate HTML outputs to reflect a series of calculations and visualizations so that we can share with clients.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#50Earlier quoted context omitted.
So you think it was a good move to scoff at someone for using a computer for their work in a way that is different from your preferences?
Notebooks are great as notebooks, but it's very well established, even in the DS community, that they are a terrible way to write maintainable, sharable, scalable code. It's not about preference, it's objectively a terrible idea to build complex workflows with notebooks. The "scoff" was in my head, the action that came out of my mouth was to help them understand how to create reusable Python modules to help them orga…
The thing with data pipelines is they have a linear execution. You start from the top and work your way down.
Notebooks do that, and even leave a trace while doing it. Table outputs, plots, etc.
It is not like a python backend that listens to events and handle them as they come, sometimes even in parallel.
For data flow, the code has an inherent direction.