Who needs MLflow when you have SQLite? • Hacker News Board

Re: Who needs MLflow when you have SQLite?

#111

Post by dataisamazing » Sun, Nov 20, 2022, 3:11 AM UTC

GeneralMayhem wrote: post #96

LeanderK wrote: post #18

I think MLflow is a good idea (very) badly executed. I would like to have a library that combines: - simple logging of (simple) metrics during and after training - simple logging of all arguments the model was created with - simple logging of a textual representation of the model - simple logging of general architecture details (number of parameters, regularisation hyperparameters, learning rate, number of epochs etc…

Yep - totally agree. I respect the attempt to introduce something that is basically an opinionated CRUD app as a central place to put your model metadata. But it's not really ready for large-scale production, or for use by teams bigger than about 5. It's kind of flaky and slow. It doesn't have namespacing. It's overly opinionated on the workflow (the way that states work, with a model version being in exactly one of…

Autologging literally does log your model metrics automatically in most cases.

Re: Who needs MLflow when you have SQLite?

#112

Post by antipaul » Mon, Nov 21, 2022, 3:20 AM UTC

Not convinced by the example. I don’t see how you can’t use standard scikit-learn for it.

First, the example doesn’t take advantage of sklearn’s built in, super simple parallelization via n_jobs

Then, the entire example could be better wrapped with sklearn’s own cross_validate() which gives you the same functionality: a table of results across experiments.

If you use a different estimator, you can easily concatenate the results into a single df

The rest is the same.

Why you need SQLite for this? (SQLite is great of course for the right use cases)

And if you're doing many orders more experiments (1000s instead of 10s) then that’s probably where MLflow is good (haven’t actually used MLflow)

Re: Who needs MLflow when you have SQLite?

#113

Post by shcheklein » Thu, Nov 24, 2022, 6:50 AM UTC

wxnx wrote: post #56

nerdponx wrote: post #51

Earlier quoted context omitted.

DVC also fills the "lightweight tracking" niche, although it relies on automatically creating Git branches as its technique for tracking experiments. I personally find that distasteful, so I don't use it specifically for experiment tracking, but the feature is there. The company behind DVC is also building a handful of other related tools, e.g. https://iterative.ai/blog/iterative-studio-model-registry

Hm, in what way do you find that DVC requires creating new branches for experiment tracking? I find the following workflow works well, for example: 1. Define steps depending on a `config.yml`. 2. Run an initial experiment (with an initial config) and commit the results. 3. Update config (preserving the alternate config and using symlinks from `config.yml` to various new configs if necessary), re-run, and commit. 4. R…

If you use `dvc exp run` you don't need to commit anything as I mentioned above. You can run multiple experiments in parallel, etc. Commit happens only / when you want to select the best result and share it with the team. But even that is optional.