Earlier quoted context omitted.
Notebooks ought to have embedded metadata, like a pyproject.toml, to list the dependencies.
https://github.com/manzt/juv
Good. This is the part I'm talking about: https://peps.python.org/pep-0723/
Representing Python notebooks as dataflow graphs
21–30 of 33 posts
Re: Representing Python notebooks as dataflow graphs
#22There are a lot of these tools to somehow "fix the reproducibility crisis of notebooks". Yet from my experience, you quickly learn to "restart kernel and run all" before sharing to make sure everything is good. All but the most novice users get caught by the "out of order cells" trap, and those will 1) not use anything that adds complexity, because by definition they are novices 2) fall in any other trap on their way…
I like marimo and I would probably use it if it weren't for years of muscle memory. That said, I don't like this reproducibility crisis story either. Notebooks are for exploration. It is okay if they are messy. If the tool I am using doesn't get in the way of my process but instead makes it fast enough then it is already doing its job. Once you are done it is up to you to let it die, make sure it is something you can…
What kind of muscle memory is holding you back? We recently added support for Jupyter-style command mode in keyboard shortcuts [1]. We're currently rewriting our VS Code extension to feel native, similar to how Jupyter feels in VS Code.
Anything else we can help with?
Re: Representing Python notebooks as dataflow graphs
#23Marimo seems really solid if you like tools like Streamlit or Observable
It certainly has some of the “widget feelings” from streamlit but the real killer feature is that you’re still always in a notebook. You can still explore with these widgets, which is a stellar experience.
Re: Representing Python notebooks as dataflow graphs
#24> You have to be very disciplined to make a Jupyter notebook that is actually reproducible This seems not necessarily very hard to me? All you have to do is keep yourself honest by actually trying to reproduce the results of the notebook when you're done: 1. Copy the notebook 2. Run from first cell in the copy 3. Check that the results are the same 4. If not the same, debug and repeat What makes it hard is when the f…
The few times I've made notebooks, I've tried to migrate code out of the notebook as soon as possible and then only import foo and run foo.bar() in the notebook. It helps to only have the top level config/layout in the notebook.
Fun detail, you can actually define functions in a marimo notebook and load them in another Python file if you want.
Needs a bit of extra config but tis a really nice feature.
Re: Representing Python notebooks as dataflow graphs
#25> You have to be very disciplined to make a Jupyter notebook that is actually reproducible This seems not necessarily very hard to me? All you have to do is keep yourself honest by actually trying to reproduce the results of the notebook when you're done: 1. Copy the notebook 2. Run from first cell in the copy 3. Check that the results are the same 4. If not the same, debug and repeat What makes it hard is when the f…
> This seems not necessarily very hard to me? All you have to do is keep yourself honest by actually trying to reproduce the results of the notebook when you're done
It's one thing when I'm relying on my own attention to detail to make sure all the intermediate results have been correctly recalculated, but it's entirely another when I have to rely on even trusted co-workers' attention to detail, much less randos on github. As a sibling comment points out, the "reproducibility crisis" numbers are very much not in favor of this approach being the right idea.
... Or you could work in a format that makes incorrect / out-of-date intermediate state impossible (or at least hard) to represent, which is (I believe) what marimo is an attempt at.
Re: Representing Python notebooks as dataflow graphs
#26Earlier quoted context omitted.
I like marimo and I would probably use it if it weren't for years of muscle memory. That said, I don't like this reproducibility crisis story either. Notebooks are for exploration. It is okay if they are messy. If the tool I am using doesn't get in the way of my process but instead makes it fast enough then it is already doing its job. Once you are done it is up to you to let it die, make sure it is something you can…
What kind of muscle memory is holding you back? We recently added support for Jupyter-style command mode in keyboard shortcuts [1]. We're currently rewriting our VS Code extension to feel native, similar to how Jupyter feels in VS Code. Anything else we can help with?
Ah that's great to hear. For me it is mostly the command mode. Is it only create / copy / paste for now? Can I also do the same for move / split / delete / undelete?
Re: Representing Python notebooks as dataflow graphs
#27As an outsider to the whole notebook ecosystem, I am absolutely gobsmacked that the representation of the notebook makes it possible to have out-of-date intermediate results. Haven't they been around for like 10+ years?
This is one of those things that is blindingly obvious to people in adjacent sectors of the industry, but maybe there just hasn't been enough cross-pollination of ideas in that direction (or in either direction).
Re: Representing Python notebooks as dataflow graphs
#28There are a lot of these tools to somehow "fix the reproducibility crisis of notebooks". Yet from my experience, you quickly learn to "restart kernel and run all" before sharing to make sure everything is good. All but the most novice users get caught by the "out of order cells" trap, and those will 1) not use anything that adds complexity, because by definition they are novices 2) fall in any other trap on their way…
Thanks for the comments. I'm the original creator of marimo. Habitually running restart and run all works okay for very lightweight notebooks, but it's a habit you need to develop, and I believe our tools should work by default. It doesn't work at all for entire categories of work, where computation is heavy and the cost of a bug is high. From the blog, you will see that reactive execution not only minimizes hidden s…
I would consider replacing my jupyterlab usage with marimo were it less opinionated about workflow - it offers a lot of benefits that aren't tied to its execution model. I like the editor/interface and the representation as python files for portability, version control, and the ability to import from other notebooks, but I have no interest in changing my workflow (in particular insofar as marimo is restricted compared to python itself). E.g., I want to be able to redefine variables and use star imports in my personal, exploratory notebooks, and I'm happy to retain responsibility for top-to-bottom executability (as in regular python scripts). I would definitely consider marimo if these restrictions could be opted out of if one has reactive execution disabled.
Re: Representing Python notebooks as dataflow graphs
#29Based on the title, I was expecting a flow editor GUI, actually.
Re: Representing Python notebooks as dataflow graphs
#30I've been using marimo since January pretty heavily, I absolutely love it and would recommend it to anyone. I run it with uv and --sandboxed which makes it much easy to share notebooks with teammates and not have to worry about limiting dependencies. Any issues I've had were were Python libraries themselves (specifically graphviz). I really like how much easier it is to reason about interactive components vs Jupyter.…
Thanks for the shoutout! We're committed to having an excellent experience for working with expensive notebooks [1]. At least for my own personal work, I find that there are many reasons to use marimo even when autorun is disabled — you still get guarantees on state, rich dataframe views, reusable functions [2], the Python file format, and more. If you have feedback on how we might improve the experience, we'd love t…
I haven’t tried marimo, so I’m not sure how it currently works, but I think instead of disabling autorun for slow and expensive computations, it sounds like it would be nicer if there were heuristics or benchmarks to automatically determine what might be slow and then execute things in varying orders lazily.