Earlier quoted context omitted.
We are working on a new paradigm on interacting with Ml training runs. A lot of the effort is now focused on very efficient experiment comparison capabilities - talking about 1000s of them. Lots of challenges on the UI and the backend. When loading TB or any other tool really with lots of experiments it's super slow and becomes useless. Also no way to do effective comparison of runs by hyperparams or other metadata o…
Sounds good (especially the performance) but ... What is the new paradigm and how does it differ from the existing paradigms? And what do you mean by "no way to do effective comparison of runs by hyperparams or other metadata on the tensorboard or MLFlow"? If you mean "you can't compare or sort a list of runs by hyperparameter or minimum loss or whatever" then MLFlow can certainly do that, so I think I'm misunderstan…
re comparison: we have always wanted to use a free open-source self-hosted tool that would let us group metrics/runs by hyperparams, experiment context(train, val, test ...) and any other adjacent info about the training runs. Be able to aggregate groups of metrics, be able to give them different styles, divide them into subplots, search through the runs easily (without regexps on super-long names) etc. As far as I checked last times no such features aren't built for those tools. This is huge motivation behind Aim.
Probably the closest to this is W&B but it's not open-source and doesn't allow to see full context of the runs while comparing them (separate module). Haven't used Losswise tbh.
We are trying to build a way that would allow to compare 1000s of ML training runs at the same time while still making the full info (context) of the runs available. This is what I meant by "new paradigm". (It turns out this is a fun problem :) ).
We have been working on Aim just a few months only (3 of us) and it's in very early stages. Most of the ideas we have aren't really shipped yet.
But it's already very useful for many RL researchers who run lots of experiments and those experiments are sensitive to hyperparameters. Aim seems to be able to handle them.
Have you checked out the live demo from the README?
Check out my blogpost on TowardsDataScience for more info on Aim (https://towardsdatascience.com/3-ways-aim-can-accelerate-you...).
Hope this info is useful and makes sense. Would be awesome to connect. I would love to learn more about your use-cases and needs in these tools. My twitter is @gevorg_s.