Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who needs MLflow when you have SQLite?

ploomber.io

Post reply on HN
Print view View original

101–110 of 113 posts

Re: Who needs MLflow when you have SQLite?

#101

Post by __mharrison__ »

BeefWellington wrote: post #95
whatever1 wrote: post #16

Earlier quoted context omitted.

I have never seen a worse documented library. Initially I thought that they were lazy, now I realize that it cannot be documented because it is a total mess of a library held together with tape. Close second is the plotly library.

I've long suspected that Pandas has taken a similar stance to e-mail scammers. Where e-mail scammers inject all kinds of broken english and bad punctuation to ensure they get their targets of choice, Pandas has broken and often inaccurate documentation in order to get only the chosen ones to work with their software. However, maybe it makes more sense that it's just a mess that's hard to document.

Do you have a specific example of this broken documentation?

Re: Who needs MLflow when you have SQLite?

#102

Post by d4rkp4ttern »

praveenhm wrote: post #36

what is alternative to MLflow other than SQLite, like Kubeflow, Metaflow?

I highly recommend ClearML for effortless experiment that just works. It does a lot more of MLOps besides experiment tracking but I haven’t used those functionalities

https://clear.ml/

I had researched and spent time with several other tools including DVC, GuildAI and MLFlow but finally settled on ClearML. WandB pricing is too aggressive for my liking (they force an annual subscription of $600 last I checked)

Re: Who needs MLflow when you have SQLite?

#103

Post by edublancas »

ralphc wrote: post #97
edublancas wrote: post #9

Earlier quoted context omitted.

Fair point. MLflow has a lot of features to cover the end-to-end dev cycle. This SQLite tracker only covers the experiment tracking part. We have another project to cover the orchestration/pipelines aspect: https://github.com/ploomber/ploomber and we have plans to work on the rest of features. For now, we're focusing on those two.

Have you looked into duckdb for the database? I'm hearing for some tasks it's faster than SQLite.

Yeah, we're also looking into it and we'll probably add it as a backend in the future!

Re: Who needs MLflow when you have SQLite?

#104

Post by tomrod »

kumare3 wrote: post #99
tomrod wrote: post #60

Earlier quoted context omitted.

Check out flyte and union.ml. No personal affiliation, just good projects in the vein of airflow/prefect/mlflow/kubeflow

@tomrod, thank you for the callout. By the way we are integrating mlflow into Flyte in a way that you do not need to start the web server to view the logs. They are available Locally and statically in Flyte Ui. Ofcourse you cal also use mlflow server

I'm having a fanboy moment. This is like tweeting about Mark Hamil or Ryan Reynolds and they tweet back.

Thanks for the great open source libraries yall!

Re: Who needs MLflow when you have SQLite?

#105

Post by nerdponx »

wxnx wrote: post #56
nerdponx wrote: post #51

Earlier quoted context omitted.

DVC also fills the "lightweight tracking" niche, although it relies on automatically creating Git branches as its technique for tracking experiments. I personally find that distasteful, so I don't use it specifically for experiment tracking, but the feature is there. The company behind DVC is also building a handful of other related tools, e.g. https://iterative.ai/blog/iterative-studio-model-registry

Hm, in what way do you find that DVC requires creating new branches for experiment tracking? I find the following workflow works well, for example: 1. Define steps depending on a `config.yml`. 2. Run an initial experiment (with an initial config) and commit the results. 3. Update config (preserving the alternate config and using symlinks from `config.yml` to various new configs if necessary), re-run, and commit. 4. R…

Right, but experiments aren't always linear. Do you really want to make a new commit for every iteration of a hyperparameter search? What if you are using a black-box optimizer that supports parallel/concurrent updates?

I don't want to use Git to track all that. I want to use Git to store the final results of running such an experiment in the same commit as the code that implemented it. I just don't like the DVC experiment workflow, but I am more than happy to use DVC for storing the fitted model(s) at the end of the run.

Re: Who needs MLflow when you have SQLite?

#107

Post by wxnx »

nerdponx wrote: post #105
wxnx wrote: post #56

Earlier quoted context omitted.

Hm, in what way do you find that DVC requires creating new branches for experiment tracking? I find the following workflow works well, for example: 1. Define steps depending on a `config.yml`. 2. Run an initial experiment (with an initial config) and commit the results. 3. Update config (preserving the alternate config and using symlinks from `config.yml` to various new configs if necessary), re-run, and commit. 4. R…

Right, but experiments aren't always linear. Do you really want to make a new commit for every iteration of a hyperparameter search? What if you are using a black-box optimizer that supports parallel/concurrent updates? I don't want to use Git to track all that. I want to use Git to store the final results of running such an experiment in the same commit as the code that implemented it. I just don't like the DVC expe…

Yeah, that's a fair point, and I agree. I don't think it's ideal.

Re: Who needs MLflow when you have SQLite?

#108

Post by avaldez_ »

chaps wrote: post #13
isoprophlex wrote: post #3

Yeah, MLFlow is a shitshow. The docs seem designed to confuse, the API makes Pandas look good and the internal data model is badly designed and exposed, as the article says. But, hordes of architects and managers who almost have a clue have been conditioned to want l and expect mlflow. And it's baked into databricks too, so for most purposes you'll be stuck with it. Props to the author for daring to challenge the sta…

"the API makes Pandas look good" It sparks joy in my heart whenever I see shade cast against pandas.

What's wrong with pandas? Honest question. I'm a bit new to ML. Also, what's the alternative?

Re: Who needs MLflow when you have SQLite?

#109

Post by j0selit0 »

I mean, come on, SQLite doesn't even support concurrency. Are people seriously considering using it in a production scenario?

If you work in a DS team where you're the only DS, then it probably suits your needs. Otherwise I can't imagine how you could achieve anything production grade

Re: Who needs MLflow when you have SQLite?

#110

Post by farbarg »

praveenhm wrote: post #36

what is alternative to MLflow other than SQLite, like Kubeflow, Metaflow?

There are a lot of tools in this space. Shameless plug to follow.

I helped build and use Disdat, which is a simple data versioning tool. It notably doesn't have the metadata capture libraries MLFlow has for different model libs, but it's meant to a lower-layer on which that can be built. Thus you won't see particulars about tracking "models" or "experiments", because models/experiments/features/intermediates are all just data thingies (or bundles in Disdat parlance). For the last 2+ years we've used Disdat to track runs and outputs of a custom distributed planning tool, and used Disdat-Luigi (an integration of Disdat with Luigi to automatically consume/produce versioned data) to manage model training and prediction pipelines (some with 10ks of artifacts). https://disdat.gitbook.io/disdat-documentation

Post reply on HN