How hard it is copy , paste and run the note book code within proper http server?
Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
21–30 of 71 posts
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#22How hard it is copy , paste and run the note book code within proper http server?
There is some utility in seeing the code and the output right below it.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#23Is this still being developed? The last commit to the main library was 5 months ago and its tied to exceptions/tests.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#24Papermill is great but has quite some limitations because it spins up a new process to run the notebook: - You cannot extract live variables (needed for testing) - Cannot use pdb for debugging - Cannot profile memory usage You can do all of that with ploomber-engine ( https://github.com/ploomber/ploomber-engine ). Disclaimer: I'm the author of this package
Not disclosed in this comment is that edublancas is > Ploomber (YC W22) co-founder.
who is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#25Earlier quoted context omitted.
Not disclosed in this comment is that edublancas is > Ploomber (YC W22) co-founder.
who is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.
What does any of this have to do with disclosure?
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#26Is this still being developed? The last commit to the main library was 5 months ago and its tied to exceptions/tests.
Seems like it's mostly died off, most people I know have moved to hosted solutions like Hex or Colab
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#27What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
We use papermill extensively, and our team is all good programmers. The difference is plots. It is a lot easier to write (and modify our existing template) to create a plot for X vs Y than it is to build and test a script that outputs e.g. a PDF.
For example, if your notebook runs into a bug, you can just run all the cells and then examine the locals after it breaks. This is extremely common when working with data (e.g. "data is missing on date X for column Y... why?").
I think most of the "real" use cases for notebooks is data analysis of various kinds, which is why a lot of people dismiss them. I wrote a blog post about this a while ago: https://rachitsingh.com/collaborating-jupyter/
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#28Jupyter notebooks are missing strict types, a linter and unit tests. When can those features be added?
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#29What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
As an MLE who comes from backend web dev, I have flip-flopped on notebooks. I initially felt that everything should be in a python script. But I see the utility in notebooks now.
For notebooks in an ML pipeline, I find that data issues are usually where things fail. Being able to run code "up to" a certain cell and create plots is invaluable. Creating reports by creating a data frame and displaying it as a cell is also super-handy.
You say, "dial some logic in", which is begging the wrong question (in my experience, at least). The logic in ML is usually very strait forward. It's about the data coming into your process and how your models are interacting with it.
Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
#30Is this still being developed? The last commit to the main library was 5 months ago and its tied to exceptions/tests.
It's a thin wrapper around notebooks. Does it really need more features? Not saying that it couldn't, but it is feature complete for what its job is.