Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who needs MLflow when you have SQLite?

ploomber.io

Post reply on HN
Print view View original

51–60 of 113 posts

Re: Who needs MLflow when you have SQLite?

#51

Post by nerdponx »

frgtpsswrdlame wrote: post #7

Wow this looks perfect for what I need right now - just a bit of lightweight tracking.

DVC also fills the "lightweight tracking" niche, although it relies on automatically creating Git branches as its technique for tracking experiments. I personally find that distasteful, so I don't use it specifically for experiment tracking, but the feature is there.

The company behind DVC is also building a handful of other related tools, e.g. https://iterative.ai/blog/iterative-studio-model-registry

Re: Who needs MLflow when you have SQLite?

#52

Post by mostdataisnice »

edublancas wrote: post #32
mostdataisnice wrote: post #21

SQLite is literally a backend for MLflow, so the argument being made really is that you should just use SQL when you can, which is kind of adjacent to any criticisms of MLflow

Is querying the underlying SQL database officially supported in MLflow? Last time I used it, it wasn't documented. I took a look at the database and it wasn't end-user friendly.

As someone replied above, it's because SQL is just 1 backend and it's weird to expose an API that only works on 1 backend. Once you have many devs working together, you need a remote server. If you have a remote abstracted backend, it needs to have a unified API surface so the same client can talk to any backend. You might argue "This interface should be SQL", and to that I would say there are many file stores (like your local file system) that are not easy to control with SQL.

Re: Who needs MLflow when you have SQLite?

#53

Post by citizenpaul »

benjaminwootton wrote: post #4

The elephant in the room with data is that we don’t need a lot of the fancy and powerful technology. SQL against a relational database gets us extraordinarily far. Add some Python scripts where we need some imperative logic and glue code, and a sprinkle of CI/CD if we really want to professionalise the work of data scientists. I think this covers the vast majority of situations. Despite being around it for some time,…

Unless your income is depending on carrying out the exact demands of some money guy that's most common phrase while using a computer is "it won't let me" and they want "big data".

Then you just suck it up and build one of the totally unnecessary big data systems that have been excreted all over the business world these days. I don't think the problem is that devs are over-engineering.

I wonder what its called, makes me think of tragedy of the commons but probably not quite right.

Re: Who needs MLflow when you have SQLite?

#54

Post by habibur »

chasil wrote: post #39
benjaminwootton wrote: post #4

The elephant in the room with data is that we don’t need a lot of the fancy and powerful technology. SQL against a relational database gets us extraordinarily far. Add some Python scripts where we need some imperative logic and glue code, and a sprinkle of CI/CD if we really want to professionalise the work of data scientists. I think this covers the vast majority of situations. Despite being around it for some time,…

The article mentions this workflow: "Let’s now execute the script multiple times, one per set of parameters, and store the results in the experiments.db SQLite database... After finishing executing the experiments, we can initialize our database (experiments.db) and explore the results." Be warned that issuing queries while DML is in process can result in SQLITE_BUSY, and the default behavior is to abort the transact…

None of these are a problem for the workload discussed.

When I am working with sqlite I am more likely accessing it from a single machine.

And in this case of ML, most likely from 1 process and by running multiple times in serial.

Re: Who needs MLflow when you have SQLite?

#55

Post by tekknolagi »

pletnes wrote: post #26
LeanderK wrote: post #18

I think MLflow is a good idea (very) badly executed. I would like to have a library that combines: - simple logging of (simple) metrics during and after training - simple logging of all arguments the model was created with - simple logging of a textual representation of the model - simple logging of general architecture details (number of parameters, regularisation hyperparameters, learning rate, number of epochs etc…

Sqlite is in python’s stdlib, so how can this be an issue? Was there no local filesystem whatsoever?

sqlite bindings are in the stdlib but not the library itself.

Re: Who needs MLflow when you have SQLite?

#56

Post by wxnx »

nerdponx wrote: post #51
frgtpsswrdlame wrote: post #7

Wow this looks perfect for what I need right now - just a bit of lightweight tracking.

DVC also fills the "lightweight tracking" niche, although it relies on automatically creating Git branches as its technique for tracking experiments. I personally find that distasteful, so I don't use it specifically for experiment tracking, but the feature is there. The company behind DVC is also building a handful of other related tools, e.g. https://iterative.ai/blog/iterative-studio-model-registry

Hm, in what way do you find that DVC requires creating new branches for experiment tracking?

I find the following workflow works well, for example:

1. Define steps depending on a `config.yml`.

2. Run an initial experiment (with an initial config) and commit the results.

3. Update config (preserving the alternate config and using symlinks from `config.yml` to various new configs if necessary), re-run, and commit.

4. Results are then all preserved in your git history.

Re: Who needs MLflow when you have SQLite?

#57

Post by shcheklein »

nerdponx wrote: post #51
frgtpsswrdlame wrote: post #7

Wow this looks perfect for what I need right now - just a bit of lightweight tracking.

DVC also fills the "lightweight tracking" niche, although it relies on automatically creating Git branches as its technique for tracking experiments. I personally find that distasteful, so I don't use it specifically for experiment tracking, but the feature is there. The company behind DVC is also building a handful of other related tools, e.g. https://iterative.ai/blog/iterative-studio-model-registry

It doesn't require creating a branch when you iterate, it requires creating a branch or commit if you want to share it with the team - see it on GitHub or in Studio. But even those lightweight iterations (https://dvc.org/doc/command-reference/exp/run) could shared as well via Git server - they won't be visible for now via UI in GH/Studio at the moment.

Happy to provide more details on how it's done. It's actually quite interesting technical thing - custom Git namespace https://iterative.ai/blog/experiment-refs

Re: Who needs MLflow when you have SQLite?

#58

Post by __mharrison__ »

whatever1 wrote: post #16
chaps wrote: post #13

Earlier quoted context omitted.

"the API makes Pandas look good" It sparks joy in my heart whenever I see shade cast against pandas.

I have never seen a worse documented library. Initially I thought that they were lazy, now I realize that it cannot be documented because it is a total mess of a library held together with tape. Close second is the plotly library.

Genuinely curious what you have against the Pandas documentation. It has some of the best docstrings I've seen.

(I also wrote a Pandas book or two... So there's that)

Re: Who needs MLflow when you have SQLite?

#60

Post by tomrod »

nerdponx wrote: post #43
smehta73 wrote: post #42

Earlier quoted context omitted.

have you used comet? it basically does everything you are asking and lot more user-friendly than MLFlow

Isn't Comet a proprietary SaaS? I like MLFlow because I can run it on my own computer if I want to.

Check out flyte and union.ml. No personal affiliation, just good projects in the vein of airflow/prefect/mlflow/kubeflow
Post reply on HN