Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

github.com

Post reply on HN
Print view View original

61–70 of 71 posts

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#61

Post by sa1 »

akshayka wrote: post #45
__mharrison__ wrote: post #44

I teach a lot using Jupyter. It is certainly possible to use SWE worst practices in Jupyter easily. I am often in front of folks who "aren't computer programmers" but need to use Python tools to be successful. One of my covert goals is to teach SWE best practices inside of notebooks. It requires a little more typing but eases the use of notebooks, refactoring, testing, moving to scripts, and using tooling like Paperm…

Have you considered using marimo notebooks? https://github.com/marimo-team/marimo marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale. marimo notebooks…

You should make a new post about the new sandbox feature.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#62

Post by wodenokoto »

akshayka wrote: post #45
__mharrison__ wrote: post #44

I teach a lot using Jupyter. It is certainly possible to use SWE worst practices in Jupyter easily. I am often in front of folks who "aren't computer programmers" but need to use Python tools to be successful. One of my covert goals is to teach SWE best practices inside of notebooks. It requires a little more typing but eases the use of notebooks, refactoring, testing, moving to scripts, and using tooling like Paperm…

Have you considered using marimo notebooks? https://github.com/marimo-team/marimo marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale. marimo notebooks…

It’s basically Pluto notebooks for Python?

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#63

Post by dayjaby »

crabbone wrote: post #59
gnulinux wrote: post #47

Earlier quoted context omitted.

Literate programming is not just "documentation + code" any more than a textbook you read about Calculus is "documentation + CalculusCode" or a novel is "documentation + plot". It goes way beyond that, using literate programming you can attach an arbitrary text that accompanies the code such that fragments of your code is simply one part of the whole text. Literate programming is not just commenting (or supercommenti…

Yes. I tried it. And, eh... it's documentation + code (you can publish code + documentation as a textbook, poem, blog post, Website just as well). No need to exaggerate. It's also very inconvenient to write, for zero benefits. It's kind of like writing prose in one language, and then translating individual pieces of it into another language, while hoping that somehow the sum will still come out OK. Some people like c…

> it's kind of like writing prose in one language, and then translating individual pieces of it into another language

That's why I'm stuck in Tolstoi's War and Peace. You have to know French to get past the first few pages.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#64

Post by crabbone »

dayjaby wrote: post #63
crabbone wrote: post #59

Earlier quoted context omitted.

Yes. I tried it. And, eh... it's documentation + code (you can publish code + documentation as a textbook, poem, blog post, Website just as well). No need to exaggerate. It's also very inconvenient to write, for zero benefits. It's kind of like writing prose in one language, and then translating individual pieces of it into another language, while hoping that somehow the sum will still come out OK. Some people like c…

> it's kind of like writing prose in one language, and then translating individual pieces of it into another language That's why I'm stuck in Tolstoi's War and Peace. You have to know French to get past the first few pages.

Ha! I had to read that in the 8th grade. At first it was very confusing, because I thought I for some reason got a book in French. But then I just skipped to the part where it started in Russian. Later, after a discussion in class, I got a vague idea that that part was some sort of a description of a ball and some high-society stuff... it wasn't at all useful for any further work we had to do on the book, so, I don't actually know what that part was about. All further reading and discussion focused on countess Olga and her thinking about the war (which the teacher claimed was the reflection of Tolstoy's own views).

But, more to the point of literal programming: it's not the only tool that wants programmers to write some sort of a plan or a sketch of the code before writing code. A much more popular technique is TDD, which, again, wants programmers to write something informally first, and then formalize it later in code. And, as with literal programming, my experience was that it's not helpful to the point of being a distraction.

There's a good reason to think that some sort of a sketch or a blueprint might be useful for the future program. It works like that in many other disciplines. Artists would make sketches before painting the picture, engineers make blueprints etc.

I think that the reason why literal programming doesn't work is because unlike a sketch or a blueprint, one has to carry it on forever (and propagate back the changes, once they are discovered) as long as the code is being worked on. It probably would've worked better if it was some sort of a plan that can be abandoned at any point, something to give the development the initial push, but not requiring any further maintenance.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#65

Post by __MatrixMan__ »

crabbone wrote: post #57
__MatrixMan__ wrote: post #39

Earlier quoted context omitted.

I sort of suspected that adding parameters was not the end of the story. My experience with this was just "make it work with papermill", so the notebooks I tested with were nice and self contained. Although it does seem like packaging dependencies and handling parameters are separate problems, so I'm not sure if papermill is to be blamed for the fact that most notebooks are not ready to be handled like a black box, e…

Jupyter is not the end of the story here. There are plenty of "extensions". These extensions go, generally, down two different ways: kernels and magic. It's not very common for Jupyter magic to be added ad hoc by users, but it typically creates a huge dependency on the environment, so no jupyenv is going to help (eg. all the workload-manager related magic to launch jobs in Slurm / OpenPBS). Kernels... well, they can…

Are we talking about the same https://github.com/tweag/jupyenv ?

It encapsulates the kernel, which encapsulates pretty much everything for the notebook, right? I haven't worked with Slurm or OpenPBS, but I think if you let nix build the images that your tasks are running in then I think you're covered for pretty much everything except things that only exist at runtime like database connections. Not a perfect black box, but close.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#66

Post by crabbone »

__MatrixMan__ wrote: post #65
crabbone wrote: post #57

Earlier quoted context omitted.

Jupyter is not the end of the story here. There are plenty of "extensions". These extensions go, generally, down two different ways: kernels and magic. It's not very common for Jupyter magic to be added ad hoc by users, but it typically creates a huge dependency on the environment, so no jupyenv is going to help (eg. all the workload-manager related magic to launch jobs in Slurm / OpenPBS). Kernels... well, they can…

Are we talking about the same https://github.com/tweag/jupyenv ? It encapsulates the kernel, which encapsulates pretty much everything for the notebook, right? I haven't worked with Slurm or OpenPBS, but I think if you let nix build the images that your tasks are running in then I think you're covered for pretty much everything except things that only exist at runtime like database connections. Not a perfect black bo…

> It encapsulates the kernel, which encapsulates pretty much everything for the notebook, right?

Not even close to everything. In real world the environment of a notebook consists of a bunch of things provided by whoever set up the lab, i.e. storage and tools.

Typical examples include setting up Lustre or Ceph in a way that it will be accessible from a notebook (but that would also involve authentication, potentially).

And, in terms of tools: a workload manager that's perhaps integrated with Jupyter to schedule notebook execution on available nodes, but also to simply run workloads. Also, just a bunch of stuff written by this or another research group. Just this week I had to install and configure some tool for arterial spin labeling, but that would be the case with any kind of research: there's plenty of stuff that researchers will rely on on, that is central to their research, but isn't directly related to Jupyter.

By and large, Jupyter is just a front-end to any particular system where research happens, it's not the system itself.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#67

Post by fluidcruft »

mcpar-land wrote: post #6

What is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.

I like using them to create reports for equipment testing. Sometimes some partner needs to be manually tweaked to get the analysis to work because there's something about the data that's difficult to generalize. Notebooks are nice because they bundle up the plots.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#68

Post by fastasucan »

crystal_revenge wrote: post #53
fifilura wrote: post #50

Earlier quoted context omitted.

The thing with data pipelines is they have a linear execution. You start from the top and work your way down. Notebooks do that, and even leave a trace while doing it. Table outputs, plots, etc. It is not like a python backend that listens to events and handle them as they come, sometimes even in parallel. For data flow, the code has an inherent direction.

> Notebooks do that, and even leave a trace while doing it. Perhaps the largest critique against notebooks is that they don't enforce a linear execution of cells. Every data scientist I know has been bitten by this at least once (not realizing they're in a stale cell that should have been updated). Sure you could solve this by automating the entire notebook ensuring top-down execution order but then why in the world…

> I can't believe I'm getting down voted for suggesting one shouldn't build complex workflows using notebooks.

I think it is more the way you express your general attitude and how you look down upon your colleague just because they have found notebooks perfectly suited for their work.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#69

Post by fastasucan »

Kalanos wrote: post #36
ziddoap wrote: post #25

Earlier quoted context omitted.

What does any of this have to do with disclosure?

calling attention to disclosure suggests bias. i'm obviously saying that i trust him not to be biased.

Not obvious at all. And no. It doesn't have to suggest bias, just a lack of disclosure.

Re: Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks

#70

Post by fastasucan »

p4ul wrote: post #13
notpushkin wrote: post #7

> Do you want to run a notebook and depending on its results, choose a particular notebook to run next? Hell no. I want to rewrite all that as a proper script or Python module.

Indeed! I feel like we as a community have taken a wrong turn with our use of notebooks. I think they have benefits in some specific use cases (e.g., teaching, demos, etc.), but otherwise, I think they mostly encourage bad practices for software development.

Luckily each person can descide what is best for their use case and does not need to follow what the "community" does.
Post reply on HN