Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Representing Python notebooks as dataflow graphs

marimo.io

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Representing Python notebooks as dataflow graphs

#11

Post by getnormality »

tastyminerals2 wrote: post #9
getnormality wrote: post #7

> You have to be very disciplined to make a Jupyter notebook that is actually reproducible This seems not necessarily very hard to me? All you have to do is keep yourself honest by actually trying to reproduce the results of the notebook when you're done: 1. Copy the notebook 2. Run from first cell in the copy 3. Check that the results are the same 4. If not the same, debug and repeat What makes it hard is when the f…

Not very hard to you, however the reproducibility numbers tell a different story. Back in the days, when we were searching for some ML model implementations in the public repos and found ipynb files in it, we skipped the repo without delving into details. Within the company data engineer research notebooks were never allowed inside a repo. Experiment, yes, but rewrite it in plain python and push.

A lot of people don't put away shopping carts, but the conclusion from that isn't that putting shopping carts away requires very high discipline. (Maybe if what is meant by "very high" is "not so low that everyone will do it", which is perhaps the point)

Re: Representing Python notebooks as dataflow graphs

#12

Post by exe34 »

getnormality wrote: post #7

> You have to be very disciplined to make a Jupyter notebook that is actually reproducible This seems not necessarily very hard to me? All you have to do is keep yourself honest by actually trying to reproduce the results of the notebook when you're done: 1. Copy the notebook 2. Run from first cell in the copy 3. Check that the results are the same 4. If not the same, debug and repeat What makes it hard is when the f…

The few times I've made notebooks, I've tried to migrate code out of the notebook as soon as possible and then only import foo and run foo.bar() in the notebook. It helps to only have the top level config/layout in the notebook.

Re: Representing Python notebooks as dataflow graphs

#13

Post by esafak »

getnormality wrote: post #7

> You have to be very disciplined to make a Jupyter notebook that is actually reproducible This seems not necessarily very hard to me? All you have to do is keep yourself honest by actually trying to reproduce the results of the notebook when you're done: 1. Copy the notebook 2. Run from first cell in the copy 3. Check that the results are the same 4. If not the same, debug and repeat What makes it hard is when the f…

Notebooks ought to have embedded metadata, like a pyproject.toml, to list the dependencies.

Re: Representing Python notebooks as dataflow graphs

#14

Post by Galanwe »

There are a lot of these tools to somehow "fix the reproducibility crisis of notebooks".

Yet from my experience, you quickly learn to "restart kernel and run all" before sharing to make sure everything is good.

All but the most novice users get caught by the "out of order cells" trap, and those will

1) not use anything that adds complexity, because by definition they are novices

2) fall in any other trap on their way because anyway that's how you learn

Thus, IMHO, these flow tools are only seen as useful by _real devs with savior syndrome_, pushing dev solution to exploratory research users, and that will never catch on.

Re: Representing Python notebooks as dataflow graphs

#15

Post by data-ottawa »

I've been using marimo since January pretty heavily, I absolutely love it and would recommend it to anyone.

I run it with uv and --sandboxed which makes it much easy to share notebooks with teammates and not have to worry about limiting dependencies. Any issues I've had were were Python libraries themselves (specifically graphviz).

I really like how much easier it is to reason about interactive components vs Jupyter. The mo.ui.altair_chart method has got me to migrate off of matplotlib because charts can be fully integrated – as you can see in the demo being able to lasso data points or scrub a chart and analyze specific time periods is awesome.

One thing which I don't like about reactive notebooks is that you have to be much more mindful of expensive and long running calculations. There are feature to help, like adding a run button, but often I end up just disabling auto-run which does reduce the value of the reactive flow. For those use cases I don't find myself using marimo over Jupyter.

I think the entire marimo team deserves a shoutout, the quality of the software is excellent, they've moved very quickly, and they have been very receptive to issues and feature suggestions.

Re: Representing Python notebooks as dataflow graphs

#16

Post by analog31 »

Galanwe wrote: post #14

There are a lot of these tools to somehow "fix the reproducibility crisis of notebooks". Yet from my experience, you quickly learn to "restart kernel and run all" before sharing to make sure everything is good. All but the most novice users get caught by the "out of order cells" trap, and those will 1) not use anything that adds complexity, because by definition they are novices 2) fall in any other trap on their way…

I compare it to day trading, where you always close out your position at the end of the day, and never leave anything running overnight. "Restart kernel and run all" when I finish a lab session. Assume that I might not come back to it for a long time, and that it won't be fresh in my mind.

It's harder when some of your cells took hours to execute, which mine don't. Then I think you should be using data files.

The tools that fix this problem seem to evoke "the cure is worse than the disease." At this point, basic Jupyter notebooks are by far the most reproducible way I've ever worked.

Re: Representing Python notebooks as dataflow graphs

#17

Post by ayhanfuat »

Galanwe wrote: post #14

There are a lot of these tools to somehow "fix the reproducibility crisis of notebooks". Yet from my experience, you quickly learn to "restart kernel and run all" before sharing to make sure everything is good. All but the most novice users get caught by the "out of order cells" trap, and those will 1) not use anything that adds complexity, because by definition they are novices 2) fall in any other trap on their way…

I like marimo and I would probably use it if it weren't for years of muscle memory. That said, I don't like this reproducibility crisis story either. Notebooks are for exploration. It is okay if they are messy. If the tool I am using doesn't get in the way of my process but instead makes it fast enough then it is already doing its job. Once you are done it is up to you to let it die, make sure it is something you can go back and iterate on it, or package it and make it usable elsewhere.

Re: Representing Python notebooks as dataflow graphs

#18

Post by kylebarron »

esafak wrote: post #13
getnormality wrote: post #7

> You have to be very disciplined to make a Jupyter notebook that is actually reproducible This seems not necessarily very hard to me? All you have to do is keep yourself honest by actually trying to reproduce the results of the notebook when you're done: 1. Copy the notebook 2. Run from first cell in the copy 3. Check that the results are the same 4. If not the same, debug and repeat What makes it hard is when the f…

Notebooks ought to have embedded metadata, like a pyproject.toml, to list the dependencies.

https://github.com/manzt/juv

Re: Representing Python notebooks as dataflow graphs

#19

Post by akshayka »

Galanwe wrote: post #14

There are a lot of these tools to somehow "fix the reproducibility crisis of notebooks". Yet from my experience, you quickly learn to "restart kernel and run all" before sharing to make sure everything is good. All but the most novice users get caught by the "out of order cells" trap, and those will 1) not use anything that adds complexity, because by definition they are novices 2) fall in any other trap on their way…

Thanks for the comments. I'm the original creator of marimo.

Habitually running restart and run all works okay for very lightweight notebooks, but it's a habit you need to develop, and I believe our tools should work by default. It doesn't work at all for entire categories of work, where computation is heavy and the cost of a bug is high.

From the blog, you will see that reactive execution not only minimizes hidden state, it also enables rapid data exploration (far more rapid than a traditional notebook), reuse as data apps, reuse as scripts, a far more intelligent module autoreloader, and much more.

marimo is not just another Jupyter extension, it's a new kind of notebook. While it may not be for you, marimo has been open source for over a year and has strong traction at many companies and universities, including by many who you may not view to be "real devs". The question of whether marimo will catch on has already been resolved :)

https://github.com/marimo-team/marimo

Re: Representing Python notebooks as dataflow graphs

#20

Post by akshayka »

data-ottawa wrote: post #15

I've been using marimo since January pretty heavily, I absolutely love it and would recommend it to anyone. I run it with uv and --sandboxed which makes it much easy to share notebooks with teammates and not have to worry about limiting dependencies. Any issues I've had were were Python libraries themselves (specifically graphviz). I really like how much easier it is to reason about interactive components vs Jupyter.…

Thanks for the shoutout!

We're committed to having an excellent experience for working with expensive notebooks [1]. At least for my own personal work, I find that there are many reasons to use marimo even when autorun is disabled — you still get guarantees on state, rich dataframe views, reusable functions [2], the Python file format, and more. If you have feedback on how we might improve the experience, we'd love to hear it.

[1] https://docs.marimo.io/guides/expensive_notebooks/

[2] https://docs.marimo.io/guides/reusing_functions/

Post reply on HN