Viewing profile — amakelov

Thanks! And thanks for sharing the pointer - I think I've seen `mltrace` at some point in the past. The tool has some similarities, but seems different from `mandala` on a philosop…

Thanks Rachit! Great running into you after all these years! Being aware of types is certainly a must in a more performance-critical implementation; this project is not at this sta…

Forgot to mention: yes, the dependency tracking is transitive, i.e. if your @op calls a @track-decorated function, which in turn calls another @track-decorated function, then both …

Thanks for sharing! This is a great project. It is quite close to the memoization part of `mandala` and I'll add it to the related work in the README. I think the similarities are:…

Great question! The versioning system does something essentially equivalent to what you describe. It currently works as follows: - When a call to an `@op` is executed, it keeps a s…

Thanks! Yes I think a caching solution like this is great for notebooks, because it makes it very cheap to re-run the whole thing (as long as you reasonably organized long-running …

Oh also totally missed the Borges mention the first time - I'm a big fan of his stories!

Thanks! Indeed, despite the fact that the main goal is to track ML experiments, the approach taken in `mandala` has a lot in common with e.g. time-travel debugging ( https://en.wik…

This blog port gives an overview of the core dependency tracking logic: https://amakelov.github.io/blog/deps/

Ah, yes, the notorious state problem in notebooks. In your project, do you find the dependencies statically or dynamically?

Great question - personally, I mostly use it from notebooks, and I think it's a great fit for that. Bundling experiment tracking with incremental computation makes a lot of sense i…

Thanks! Indeed, the ultimate (but very ambitious from the point of view of coordination and infrastructure) vision would be to build the "planetary computer" where everyone can con…

In order, 1. Yes, you can choose to create a persistent storage by passing `db_path` to `Storage()`. The current implementation is just an SQLite file. To run on many machines, you…

`mandala` is a framework I wrote to automate tracking ML experiments for my research. It differs from other experiment tracking tools by making persistence, query and versioning lo…

This is neat and self-contained! But as someone running experiments with a high degree of interactivity, I often have an orthogonal requirement: add more computations to the same c…

I see two concerns here: - inputs/outputs being high volume: the inputs/outputs that are large are often also things that don't change over the course of a project (e.g. a dataset …

Author here - thank you for the compliment! Always happy to answer any questions (the docs are admittedly quite sparse) and hear about what people like/dislike about the library!

The dependency tracking and the graph/SQL thing are largely independent pieces. The graph/SQL thing is used to query the memoization tables of the memoized functions by "joining th…

Hi, author here. Sorry about the confusion - this blog post's intention was to give a more programming-language-themed introduction to the project (discussion on r/programminglangu…