Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Mandala – Automatically save, query and version Python computations

github.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Show HN: Mandala – Automatically save, query and version Python computations

#11

Post by ledauphin »

We have a framework at to do memoization as well as distributed compute - in fact the memoization was mostly a happy side effect of the need to transfer serialized function arguments to other machines.

Your addition of code/runtime dependencies intrigues me. I will probably take a look at your code to try to understand this better.

I somehow doubt there's enough overlap for us to open source our work and try to merge with yours, but it's really cool to see other people working on similar concepts. I predict we'll see a lot more frameworks like these that lean on mathematical principles like functional purity in the future.

Re: Show HN: Mandala – Automatically save, query and version Python computations

#12

Post by amakelov »

ledauphin wrote: post #11

We have a framework at to do memoization as well as distributed compute - in fact the memoization was mostly a happy side effect of the need to transfer serialized function arguments to other machines. Your addition of code/runtime dependencies intrigues me. I will probably take a look at your code to try to understand this better. I somehow doubt there's enough overlap for us to open source our work and try to merge…

This blog port gives an overview of the core dependency tracking logic: https://amakelov.github.io/blog/deps/

Re: Show HN: Mandala – Automatically save, query and version Python computations

#13

Post by ledauphin »

amakelov wrote: post #12
ledauphin wrote: post #11

We have a framework at to do memoization as well as distributed compute - in fact the memoization was mostly a happy side effect of the need to transfer serialized function arguments to other machines. Your addition of code/runtime dependencies intrigues me. I will probably take a look at your code to try to understand this better. I somehow doubt there's enough overlap for us to open source our work and try to merge…

This blog port gives an overview of the core dependency tracking logic: https://amakelov.github.io/blog/deps/

thank you!

Re: Show HN: Mandala – Automatically save, query and version Python computations

#14

Post by vrtnis »

amakelov wrote: post #10
vrtnis wrote: post #9

Very cool! looking forward to trying it out - the graphs reminded me of a toy project I'd done a while back to better understand deterministic and reproducible execution in python as seen in marimo.io notebooks https://github.com/vrtnis/python-notebook-simulator

Ah, yes, the notorious state problem in notebooks. In your project, do you find the dependencies statically or dynamically?

Statically - basically just parsing the code into an AST and then walking through the tree to collect information about variable usage and definitions.

Re: Show HN: Mandala – Automatically save, query and version Python computations

#15

Post by culebron21 »

Congratulations, good job! The chaos of notebooks needs some tracking indeed.

7 years ago I made a project with 100 calculation dependencies, (in Python & SQL scripts) and the only thing that allowed not to loose track was Makefile + GraphViz.

I wanted to make something similar in Rust -- a static visualized of dependencies between structs. Things turned out way harder than expected.

Re: Show HN: Mandala – Automatically save, query and version Python computations

#16

Post by chudi »

This is great! Really good work. It reminds me of working in a SmallTalk environment and coding inside the debugger while an exception is being thrown and restarting the computation.

I believe that this path can be supported as it is right now, and the next step would be to store a computation on some server. If an uncaught exception is raised, store all the computation along with the state, transfer it to your local machine, and restore the state of the machine as it was when the exception was thrown. This way, you can debug the state of the program with all the live variables as it was being run.

Re: Show HN: Mandala – Automatically save, query and version Python computations

#17

Post by amakelov »

chudi wrote: post #16

This is great! Really good work. It reminds me of working in a SmallTalk environment and coding inside the debugger while an exception is being thrown and restarting the computation. I believe that this path can be supported as it is right now, and the next step would be to store a computation on some server. If an uncaught exception is raised, store all the computation along with the state, transfer it to your local…

Thanks! Indeed, despite the fact that the main goal is to track ML experiments, the approach taken in `mandala` has a lot in common with e.g. time-travel debugging (https://en.wikipedia.org/wiki/Time_travel_debugging).

In reality, there are many similarities between experiment tracking, debugging, and high-level computation graphs - they're all different ways of getting a handle on what a program did when it was ran.

Re: Show HN: Mandala – Automatically save, query and version Python computations

#18

Post by amakelov »

taliesinb wrote: post #4

Cool! Looks pretty professional. I explored a similar idea once (also implemented in Python, via decorators) to help speed up some neuroscience research that involved a lot of hyperparameter sweeps. It's named after a Borges story about a man cursed to remember everything: https://github.com/taliesinb/funes Maybe one day we'll have a global version of this, where all non-private computations are cached on a global di…

Oh also totally missed the Borges mention the first time - I'm a big fan of his stories!

Re: Show HN: Mandala – Automatically save, query and version Python computations

#19

Post by pbronez »

How well does Mandala extend beyond the Numpy ecosystem?

I’m experimenting with Python CAD programming using the CadQuery and Build123d libraries. I’d like to speed up iteration time and intelligent caching would help.

These libraries are pretty opinionated, which make it a bit challenging to imagine how to squeeze cache decorators in there. They have a couple different APIs, all of which ultimately use the Open CASCADE (OCCT) kernel via https://github.com/CadQuery/OCP

CadQuery is a Fluent programming design that relies on long Method Chains [0]. It also has an experimental Free Function API [1].

Build123d iterates on CadQuery [2] with the goal of integrating better with the Python programming language. It has two supported APIs. The Builder API uses Python’s context manager (‘with’ blocks) heavily. The secondary Algebraic API is more functional, using arithmetic operators to define geometric operations [3].

The simplest way to integrate Mandala would probably be to use Build123d’s Algebraic API, wrapping subassemblies in functions decorated with @op.

However, it would be even better to proactively cache function/argument pairs provided by the underlying APIs. For example, if I change 50% of the edges passed to a Fillet() call, it would be nice to have it complete in half the time. I guess this would require me to fork the underlying library and integrate Mandala at that level.

[0] https://cadquery.readthedocs.io/en/latest/intro.html

[1] https://cadquery.readthedocs.io/en/latest/free-func.html

[2] https://build123d.readthedocs.io/en/latest/introduction.html...

[3] https://build123d.readthedocs.io/en/latest/key_concepts_alge...

Re: Show HN: Mandala – Automatically save, query and version Python computations

#20

Post by amakelov »

culebron21 wrote: post #15

Congratulations, good job! The chaos of notebooks needs some tracking indeed. 7 years ago I made a project with 100 calculation dependencies, (in Python & SQL scripts) and the only thing that allowed not to loose track was Makefile + GraphViz. I wanted to make something similar in Rust -- a static visualized of dependencies between structs. Things turned out way harder than expected.

Thanks! Yes I think a caching solution like this is great for notebooks, because it makes it very cheap to re-run the whole thing (as long as you reasonably organized long-running computations into `@op`s), overcoming the notorious state problem.

Graphviz is indeed a lifesaver; and you can similarly think of `mandala` as a "build system for Python objects" (with all the cool and uncool things that come with that; serializing arbitrary Python objects with strong guarantees is hard https://amakelov.github.io/mandala/tutorials/gotchas/).

I've no experience with rust, but I'd be curious to hear about the difficulties that came up. I'd expect to see some overlap with Python!

Post reply on HN