Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned reinventing the Python notebook

marimo.io

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Lessons learned reinventing the Python notebook

#11

Post by dan-g »

tiledjinn wrote: post #9
parsimo2010 wrote: post #7

Earlier quoted context omitted.

Then don’t use it. This isn’t a requirement to use Python. It’s just one of the alternatives including notebooks, a text editor, or an IDE. But notebooks are very popular in some fields, and it’s nice to see new things.

no, it's a failed concept. frequently you'll find effectively shell scripts that couple workflows to some graphical environment for spurious reasons -- except they're baked with bad assumptions about state in that environment as outlined in the article. if you want knuth's literate programming, just comment liberally per that guidance. or carry forward with them and print your retractions later. vOv

I’m not sure what your definition of “failed” is, but notebooks of various kinds have been bringing value and accessibility to various different programming languages for a while now. If it was a “failed concept” I don’t think we’d see as much adoption, let alone new iterations and advancements in the space.

Just because you don’t use something doesn’t mean it’s not worth something to someone else (or in this case, a lot of people), despite its shortcomings.

Re: Lessons learned reinventing the Python notebook

#12

Post by lmeyerov »

dleeftink wrote: post #4

How about lessons learned from other reactive notebook environments that have been around for a while? If you are 'reinventing' something, I wonder why no mentions are made of the paths that have been laid before (Observable, Quarto, Enso, etc.).

Their GitHub does ;-)

I enjoyed the article. We are doing a more genAI-oriented notebook env variant for louie.ai aimed at database/API users preferring natural-language-first vs typical Jupyter/observable/hex/etc python wrappers, and for their automatic reactivity portion, a lot of similarity here with the road we went down. I appreciate seeing some of that reasoning laid out. Especially neat is the use of static analysis of global assignments vs the typical input/output variable syntactic convention. Few go so fine-grained (enso, frp, ...).

My personal experiences with automatic reactivity (OSS'ing the first web frp system, and many after) and last 10 years as a heavy Jupyter notebooks user with many enterprise/gov/tech DS+operational teams actually steered us down a different path. We separate out the auto-reactivity to only auto-trigger when switched into a 'live' dashboarding view. By default, when in the authoring notebook view, we stick to Jupyter-style manually triggered run / runall. When authoring, the notebook accidently running a bunch of database or OpenAI or bigquery calls when you didn't intend can be... problematic. The underlying cell-based reactivity model we went with is largely the same, but when an analyst is dealing with big database calls and returns, a world of a difference in UX.

Fwiw, if anyone uses Splunk/databricks/OpenSearch/etc and curious about a team genAI-first learning system with auto-viz-reactivity, would love to chat!

Re: Lessons learned reinventing the Python notebook

#13

Post by wokwokwok »

> marimo makes it seamless to go from prototype to production.

No, it doesn’t.

In fact, no tool; not retool, not any of no-code solutions, free or paid, lets you take a hacked up prototype and seamlessly put in into production.

In fact, even claiming that it does makes me question the other (superficially quite laudable) things it claims to do; eg. plain python files.

Why do people go down these rabbit holes?

You can’t run prototype notebooks for your production workloads.

It’s not because it’s fundamentally impossible to use a notebook as a platform for robust production code.

…it’s because prototyping means quick hacks, quick dirty unmaintainable hacks.

> Run your notebook as a web app, with Python code hidden and uneditable

Why? Stop. No. Don’t make your prototype hacked up notebook that has probably got direct system access public.

You at least are password protecting this right? No? (https://github.com/marimo-team/marimo/issues/395)

What part of don’t accept every damn idea as a feature did you not take to heart?

-_-

There are a lot of ideas in this project. A LOT of ideas. Some of them are good. Some of them are fully implemented.

Re: Lessons learned reinventing the Python notebook

#14

Post by jrajav »

tiledjinn wrote: post #9
parsimo2010 wrote: post #7

Earlier quoted context omitted.

Then don’t use it. This isn’t a requirement to use Python. It’s just one of the alternatives including notebooks, a text editor, or an IDE. But notebooks are very popular in some fields, and it’s nice to see new things.

no, it's a failed concept. frequently you'll find effectively shell scripts that couple workflows to some graphical environment for spurious reasons -- except they're baked with bad assumptions about state in that environment as outlined in the article. if you want knuth's literate programming, just comment liberally per that guidance. or carry forward with them and print your retractions later. vOv

Generally notebooks are used specifically when statefulness gets in the way, and an interactive feedback loop is the entire end goal, not the means. Being able to iterate on operations at a cell level with reactivity is totally game changing for such use cases. If you don't get where that would be valuable, you probably just haven't encountered such a use case yet.

Re: Lessons learned reinventing the Python notebook

#16

Post by taeric »

I find it so weird to see Jupyter being the "traditional" option in this discussion.

Kudos for the shout out to excel being the main "notebook" most people have experience with.

I question the choice of Python as the format. Agreed that Json is terrible. But Python will have similar problems. And a notebook is a marked up document. Using a markup language is a natural choice. Delimit areas of particular syntax and you are off to the races. (Yes, I am fond of org mode.)

Re: Lessons learned reinventing the Python notebook

#17

Post by Kalanos »

fbdab103 wrote: post #10
dleeftink wrote: post #4

How about lessons learned from other reactive notebook environments that have been around for a while? If you are 'reinventing' something, I wonder why no mentions are made of the paths that have been laid before (Observable, Quarto, Enso, etc.).

No mention of Julia's Pluto seems like a real miss. I am Python day-to-day, but the Pluto demos I have seen are very attractive. I think Pluto also lets you reassign a variable.

i spent time w pluto. it's not at parity

Re: Lessons learned reinventing the Python notebook

#18

Post by Kalanos »

as a daily jupyter user, i like this .py DAG approach a lot - more so for visualizing complex workflows than addressing a potentially corrupt hidden state.

there isn't much *data science functionality* integrated into jupyter core apart from displaying static dataframes (nor is their internal desire to add it), and jupyter *extensions/widgets* should really be built with dash/streamlit, so i'd encourage you to make an ecosystem.

having spent a lot of time with airflow recently, it's designed for scheduling recurring jobs, not *batch processing.* meanwhile dagster is not meant for computationally intensive/ long-running jobs and the dataset-centric approach is offputting

overengineering = since you made the cells functions already, it would be cool if you could make multiprocessing/multithreading an option in the cell UI and help keep track of multi-state variables/execution

happy to give feedback/ user interview = linkedin.com/in/laynesadler/

Re: Lessons learned reinventing the Python notebook

#19

Post by mistrial9 »

jrajav wrote: post #14
tiledjinn wrote: post #9

Earlier quoted context omitted.

no, it's a failed concept. frequently you'll find effectively shell scripts that couple workflows to some graphical environment for spurious reasons -- except they're baked with bad assumptions about state in that environment as outlined in the article. if you want knuth's literate programming, just comment liberally per that guidance. or carry forward with them and print your retractions later. vOv

Generally notebooks are used specifically when statefulness gets in the way, and an interactive feedback loop is the entire end goal, not the means. Being able to iterate on operations at a cell level with reactivity is totally game changing for such use cases. If you don't get where that would be valuable, you probably just haven't encountered such a use case yet.

yes and, the strong interactive graphics inline are important for a lot of learning situations

Re: Lessons learned reinventing the Python notebook

#20

Post by akshayka »

fbdab103 wrote: post #10
dleeftink wrote: post #4

How about lessons learned from other reactive notebook environments that have been around for a while? If you are 'reinventing' something, I wonder why no mentions are made of the paths that have been laid before (Observable, Quarto, Enso, etc.).

No mention of Julia's Pluto seems like a real miss. I am Python day-to-day, but the Pluto demos I have seen are very attractive. I think Pluto also lets you reassign a variable.

I'm a marimo author. Pluto is amazing, and definitely one of our biggest inspirations. As others have mentioned, we've given shout outs to Pluto.JL in most of our copy; oversight to not include it here.
Post reply on HN