Why Metaflow?

#1

Post by savin-goyal » Wed, Jul 30, 2025, 1:09 AM UTC

Re: Why Metaflow?

#2

Post by sito42 » Wed, Jul 30, 2025, 2:15 AM UTC

I don't understand why this framework has become so popular. You can't instantiate or program around `FlowSpec`s. The type safety is terrible (you're supposed to just haphazardly assign to `self` whenever you need to persist something across steps). `step`s just feel like glorified jupyter cells.

There's been so much great innovation in the ML ops space in the last few years and metaflow puts you in a straightjacket that prevents you from tapping into any of it.

Re: Why Metaflow?

#3

Post by constGard » Wed, Jul 30, 2025, 5:28 AM UTC

sito42 wrote: post #2

I don't understand why this framework has become so popular. You can't instantiate or program around `FlowSpec`s. The type safety is terrible (you're supposed to just haphazardly assign to `self` whenever you need to persist something across steps). `step`s just feel like glorified jupyter cells. There's been so much great innovation in the ML ops space in the last few years and metaflow puts you in a straightjacket…

Off the top of your head, could you describe a good/great ml ops infra setup? I've been reading what I can about state of the art ml ops after not being in the space for a few years and metaflow comes up a lot.

Re: Why Metaflow?

#4

Post by knowsuchagency » Wed, Jul 30, 2025, 9:33 PM UTC

constGard wrote: post #3

sito42 wrote: post #2

I don't understand why this framework has become so popular. You can't instantiate or program around `FlowSpec`s. The type safety is terrible (you're supposed to just haphazardly assign to `self` whenever you need to persist something across steps). `step`s just feel like glorified jupyter cells. There's been so much great innovation in the ML ops space in the last few years and metaflow puts you in a straightjacket…

Off the top of your head, could you describe a good/great ml ops infra setup? I've been reading what I can about state of the art ml ops after not being in the space for a few years and metaflow comes up a lot.

Nothing touches https://windmill.dev. There are some rare instances where you still may want Airflow, but even so, windmill and airflow complement each other well. If you need notebooks, I recommend https://docs.marimo.io/