Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who needs MLflow when you have SQLite?

ploomber.io

Post reply on HN
Print view View original

61–70 of 113 posts

Re: Who needs MLflow when you have SQLite?

#61

Post by tomrod »

citizenpaul wrote: post #53
benjaminwootton wrote: post #4

The elephant in the room with data is that we don’t need a lot of the fancy and powerful technology. SQL against a relational database gets us extraordinarily far. Add some Python scripts where we need some imperative logic and glue code, and a sprinkle of CI/CD if we really want to professionalise the work of data scientists. I think this covers the vast majority of situations. Despite being around it for some time,…

Unless your income is depending on carrying out the exact demands of some money guy that's most common phrase while using a computer is "it won't let me" and they want "big data". Then you just suck it up and build one of the totally unnecessary big data systems that have been excreted all over the business world these days. I don't think the problem is that devs are over-engineering. I wonder what its called, makes…

Hierarchy on bueracracies, by Jean Tirole. I know because this was the phenomenon I wanted to study in grad school only to find he scooped me (on this an several items) by several decades.

Edit: Tirole, Jean. "Hierarchies and bureaucracies: On the role of collusion in organizations." JL Econ. & Org. 2 (1986): 181.

Re: Who needs MLflow when you have SQLite?

#62

Post by YetAnotherNick »

tomrod wrote: post #60
nerdponx wrote: post #43

Earlier quoted context omitted.

Isn't Comet a proprietary SaaS? I like MLFlow because I can run it on my own computer if I want to.

Check out flyte and union.ml. No personal affiliation, just good projects in the vein of airflow/prefect/mlflow/kubeflow

I really like guild.ai. The best thing is that their developers assumed people to be lazy and automatically makes flag for global variables and track them.

Re: Who needs MLflow when you have SQLite?

#63

Post by akdor1154 »

dachryn wrote: post #14
idomi wrote: post #11

Earlier quoted context omitted.

How many data scientists that use Databricks for modeling do you know?

its forced upon many of them that are in finance, banking, insurance, ... Mainly because those tend to run on Microsoft Azure, which has no decent analytics offering, and are pushing Databricks extremely hard. The CTO or whatever just pushes databricks. On paper it checks all the boxes. Mlops, notebooks, experiment management. It just does all of those things very badly, but the exec doesn't care. They only care abou…

What would you go with instead for collaborative notebooks?

I ask because normally I tend pretty strongly towards the "NO just let the DSes/analysts work how they want to", which in this case would be running Jupyter locally. However DBr's notebooks seem genuinely useful.

Is your issue "but I don't need Spark" or "i wanna code in a python project, not a notebook?", or something else?

Imo if DBr cut their wedding to Spark and provided a Python-only nb environment they'd have a killer offering on their hands.

Re: Who needs MLflow when you have SQLite?

#65

Post by claytonjy »

nerdponx wrote: post #47
whatever1 wrote: post #16

Earlier quoted context omitted.

I have never seen a worse documented library. Initially I thought that they were lazy, now I realize that it cannot be documented because it is a total mess of a library held together with tape. Close second is the plotly library.

The Pandas documentation has improved quite a bit. Last I checked, the only part of the reference docs with a big gap was the description of "extension arrays" and accessors. The user guide material absolutely needs work, and the examples in the reference docs tend to be a little contrived. But I absolutely have seen worse-documented libraries, such as Gunicorn and Pydantic.

I'm surprised to see Pydantic in here; I've used Pandas and Pydantic both quite a lot, and have found the Pydantic docs to be quite good! Also a much smaller library with a saner API, and thus easier to document well.

Re: Who needs MLflow when you have SQLite?

#66

Post by 333luke »

whatever1 wrote: post #16
chaps wrote: post #13

Earlier quoted context omitted.

"the API makes Pandas look good" It sparks joy in my heart whenever I see shade cast against pandas.

I have never seen a worse documented library. Initially I thought that they were lazy, now I realize that it cannot be documented because it is a total mess of a library held together with tape. Close second is the plotly library.

What makes the documentation so bad in your opinion? I’m not arguing but curious since I use pandas all day at my job and can’t think of any times the docs weren’t clear to me. (Plotly I have had some annoying times with!)

Re: Who needs MLflow when you have SQLite?

#67

Post by tomrod »

nerdponx wrote: post #44
tomrod wrote: post #37

Earlier quoted context omitted.

I've heavily used R several times. I agree that it is a trash language and that, outside that many frontier academic ideas are available and some plotting preferences are solidly prescriptive, it should be thrown into the trash bin. Python, Julia when it gets its druthers for TTFP, Octave, Fortran, C, and eventually Rust. These are the tools I've found in use over and over and over again across business, government,…

Try to separate the language from its standard library. Neither one is "trash". I agree that the standard library is what you might call "a chaotic disorganized mess".

I'm not emotionally invested in tools so am happy to identify the user experience and operational experience as "trash."

"Trash", despite its connotations of lacking value, is really just a chaotic disorganized mess of something made by artifice with dubious reclaim/reuse/recycle value. Being a subjective assessment, it is natural that one person's trash is a treasure to another.

Re: Who needs MLflow when you have SQLite?

#68

Post by imachine1980_ »

tekknolagi wrote: post #55
pletnes wrote: post #26

Earlier quoted context omitted.

Sqlite is in python’s stdlib, so how can this be an issue? Was there no local filesystem whatsoever?

sqlite bindings are in the stdlib but not the library itself.

im asking from ignorance, what the difference in effect this context of not having the library itself?

Re: Who needs MLflow when you have SQLite?

#69

Post by morelisp »

citizenpaul wrote: post #53
benjaminwootton wrote: post #4

The elephant in the room with data is that we don’t need a lot of the fancy and powerful technology. SQL against a relational database gets us extraordinarily far. Add some Python scripts where we need some imperative logic and glue code, and a sprinkle of CI/CD if we really want to professionalise the work of data scientists. I think this covers the vast majority of situations. Despite being around it for some time,…

Unless your income is depending on carrying out the exact demands of some money guy that's most common phrase while using a computer is "it won't let me" and they want "big data". Then you just suck it up and build one of the totally unnecessary big data systems that have been excreted all over the business world these days. I don't think the problem is that devs are over-engineering. I wonder what its called, makes…

Maybe like 20 years ago you were right but today there's a generation that's been working for 10 years on systems built like that. They don't know any better, and in most cases nobody is around to teach them otherwise.

Re: Who needs MLflow when you have SQLite?

#70

Post by funklute »

imachine1980_ wrote: post #68
tekknolagi wrote: post #55

Earlier quoted context omitted.

sqlite bindings are in the stdlib but not the library itself.

im asking from ignorance, what the difference in effect this context of not having the library itself?

Using the bindings is only possible if the library itself is already installed (since the bindings directly make use of the library, under the hood).
Post reply on HN