Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

github.com

Post reply on HN
Print view View original

51–60 of 71 posts

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#51

Post by cycomanic »

akshayka wrote: post #45
__mharrison__ wrote: post #44

I teach a lot using Jupyter. It is certainly possible to use SWE worst practices in Jupyter easily. I am often in front of folks who "aren't computer programmers" but need to use Python tools to be successful. One of my covert goals is to teach SWE best practices inside of notebooks. It requires a little more typing but eases the use of notebooks, refactoring, testing, moving to scripts, and using tooling like Paperm…

Have you considered using marimo notebooks? https://github.com/marimo-team/marimo marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale. marimo notebooks…

There is also jupytext which converts Jupyter notebooks on the fly to a number of different formats (Markdown, python,...). It's at the core of the Jupyterbook project IIRC and IMO the best method to use Jupyter with git.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#52

Post by __mharrison__ »

cycomanic wrote: post #51
akshayka wrote: post #45

Earlier quoted context omitted.

Have you considered using marimo notebooks? https://github.com/marimo-team/marimo marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale. marimo notebooks…

There is also jupytext which converts Jupyter notebooks on the fly to a number of different formats (Markdown, python,...). It's at the core of the Jupyterbook project IIRC and IMO the best method to use Jupyter with git.

I use Jupytext (and my own conversion utilities) all the time. I write my books inside of Jupyter these days.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#53

Post by crystal_revenge »

fifilura wrote: post #50
crystal_revenge wrote: post #46

Earlier quoted context omitted.

Notebooks are great as notebooks, but it's very well established, even in the DS community, that they are a terrible way to write maintainable, sharable, scalable code. It's not about preference, it's objectively a terrible idea to build complex workflows with notebooks. The "scoff" was in my head, the action that came out of my mouth was to help them understand how to create reusable Python modules to help them orga…

The thing with data pipelines is they have a linear execution. You start from the top and work your way down. Notebooks do that, and even leave a trace while doing it. Table outputs, plots, etc. It is not like a python backend that listens to events and handle them as they come, sometimes even in parallel. For data flow, the code has an inherent direction.

> Notebooks do that, and even leave a trace while doing it.

Perhaps the largest critique against notebooks is that they don't enforce a linear execution of cells. Every data scientist I know has been bitten by this at least once (not realizing they're in a stale cell that should have been updated).

Sure you could solve this by automating the entire notebook ensuring top-down execution order but then why in the world are you using a notebook like this? There is no case I can think of where this would be remotely better than just pulling out the code into shared libraries.

I've worked on a wide range of data science teams in my career and by far the most productive ones are the ones that have large shared libraries and have a process in place for getting code out of notebooks and into a proper production pipeline.

Normally I'm the person defending notebooks since there's a growing number of people who outright don't want to see them used ever. But they do have their place, as notebooks. I can't believe I'm getting down voted for suggesting one shouldn't build complex workflows using notebooks.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#54

Post by zelphirkalt »

gnulinux wrote: post #17
mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

It's a literate programming tool. If you find literate programming useful (such as Donald Knuth's Latex) then you can write a Jupyter notebook, add text, add latex, titles, paragraphs, explanations, stories and attach code too. Then, you can just run it. I know that this sounds pretty rare but this is mostly how I write code (not in Jupyter notebook, I use Markdown instead and write code in a combination of Obsidian…

Does it support more of literate programming than the small amount of features, that normal Jupyter notebook supports?

I always wish they would take a hint from Emacs org mode and make notebooks more useful for development.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#55

Post by gnulinux »

zelphirkalt wrote: post #54
gnulinux wrote: post #17

Earlier quoted context omitted.

It's a literate programming tool. If you find literate programming useful (such as Donald Knuth's Latex) then you can write a Jupyter notebook, add text, add latex, titles, paragraphs, explanations, stories and attach code too. Then, you can just run it. I know that this sounds pretty rare but this is mostly how I write code (not in Jupyter notebook, I use Markdown instead and write code in a combination of Obsidian…

Does it support more of literate programming than the small amount of features, that normal Jupyter notebook supports? I always wish they would take a hint from Emacs org mode and make notebooks more useful for development.

No it supports less actually. Obsidian is only a markdown editor, it does allow you to edit code fragments like code (so there is basic code highlighting, auto-tabbing etc) but that's it. I personally find this a lot easier in some cases. I find that sometimes if the code is too complicated that you need anything more than just "seeing" you probably need to break it further down to its atomic elements. For certain kinds of development, I do find myself needing to be in "programming groove" then I use Emacs. But other times, I accompany the code with a story and/or technical description so it feels like the end goal is to write the document, and not the code. Executable code is just a artifact that comes with it. It's definitely a niche application as far e.g. the industry goes.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#56

Post by throwpoaster »

edublancas wrote: post #5
throwpoaster wrote: post #4

Earlier quoted context omitted.

iirc, a few years back I was able to do all of these things with the Papermill IPython runtime. Papermill is great, but yes: lots of room to hack on it and make it better.

has papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.

It’s been a while but you do it with a custom kernel and maybe some entry point tweaks. IIRC.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#57

Post by crabbone »

__MatrixMan__ wrote: post #39
crabbone wrote: post #33

Earlier quoted context omitted.

I've been that person, and no it doesn't. It makes my life suck, if I have to include a notebook instead of an actual program in a larger program. Notebooks don't compose well, they are too dependent on the specifics of the environment in which they were launched, they have excessive source code that's also machine-generated and is hard to work with for humans. As a stop-gap solution, for cases like a single presenta…

I sort of suspected that adding parameters was not the end of the story. My experience with this was just "make it work with papermill", so the notebooks I tested with were nice and self contained. Although it does seem like packaging dependencies and handling parameters are separate problems, so I'm not sure if papermill is to be blamed for the fact that most notebooks are not ready to be handled like a black box, e…

Jupyter is not the end of the story here. There are plenty of "extensions". These extensions go, generally, down two different ways: kernels and magic.

It's not very common for Jupyter magic to be added ad hoc by users, but it typically creates a huge dependency on the environment, so no jupyenv is going to help (eg. all the workload-manager related magic to launch jobs in Slurm / OpenPBS).

Kernels... well, they can do all sorts of things... beyond your wildest dreams and imagination. And, unlike magic, they are readily available for the end-user to mess with. And, of course, there are a bunch of pre-packaged ones, supplied by all sorts of vendors who want, in this way, to promote their tech. Say, stuff like running Jupyter over Kubernetes with Ceph volumes exposed to the notebook. There's no easy way of making this into a "module" / "black box" that can be combined with some other Python code. It needs a ton of infra code to support this, if it's meant to be somewhat stand-alone.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#58

Post by crabbone »

abdullahkhalids wrote: post #34
crabbone wrote: post #31

Earlier quoted context omitted.

I have to disagree... Literate programming is still programming: it produces programs (but with an extra effort of writing documentation up-front). Jupyter is a tool to do some exploratory interactive programming. Most notebooks I've seen in my life (probably thousands at this point) are worthless as complete programs. They are more akin to shell sessions, which, for the most part, I wouldn't care storing for later.…

> Don’t get discouraged because there’s a lot of mechanical work to writing. There is, and you can’t get out of it. I rewrote A Farewell to Arms at least fifty times. You’ve got to work it over. The first draft of anything is shit. Ernest Hemingway This is how all intellectual work proceeds. Most of the stuff you write is crap. After many iterations you produce one that is good enough for others. Should we take away…

I think, you completely missed the point... I compared Jupyter notebooks to shell sessions: it doesn't make them bad (they are, however, but for a different reason). I don't think that shell sessions are bad. The point I'm making is that Jupyter notebooks aren't suitable for being independent modules inside a larger program (and so are shell sessions). The alternative is obvious: just write the program.

Can you possibly make Jupyter notebook act like a module in a program? -- with a lot of effort and determination, yes. Should you be doing this, especially since the alternative is very accessible and produces far superior results? -- Of course no.

Using your metaphor, I'm not arguing for taking the typewriter away from the not-so-good writers. I'm arguing that maybe they can use a computer with a word processor, so that they don't waste so much paper.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#59

Post by crabbone »

gnulinux wrote: post #47
crabbone wrote: post #31

Earlier quoted context omitted.

I have to disagree... Literate programming is still programming: it produces programs (but with an extra effort of writing documentation up-front). Jupyter is a tool to do some exploratory interactive programming. Most notebooks I've seen in my life (probably thousands at this point) are worthless as complete programs. They are more akin to shell sessions, which, for the most part, I wouldn't care storing for later.…

Literate programming is not just "documentation + code" any more than a textbook you read about Calculus is "documentation + CalculusCode" or a novel is "documentation + plot". It goes way beyond that, using literate programming you can attach an arbitrary text that accompanies the code such that fragments of your code is simply one part of the whole text. Literate programming is not just commenting (or supercommenti…

Yes. I tried it. And, eh... it's documentation + code (you can publish code + documentation as a textbook, poem, blog post, Website just as well). No need to exaggerate. It's also very inconvenient to write, for zero benefits. It's kind of like writing prose in one language, and then translating individual pieces of it into another language, while hoping that somehow the sum will still come out OK.

Some people like challenge in their lives... and I don't blame them. For sport, I would also rewrite some silly programs in languages I never intend to use, or do some code-golfing etc. Literate programming belongs in this general area of making extra effort to accomplish something that would've been trivial to do in a much simpler way.

Post reply on HN