Live data from Hacker News

Why we do machine learning engineering with YAML, not notebooks

towardsdatascience.com

11–20 of 31 posts

Re: Why we do machine learning engineering with YAML, not notebooks

#11
For .ipynb notebooks, I highly recommend using nbstripout [0] to strip the Jupiter output before committing the notebooks to the repository (thus making the diffs sane).

You can also set it up as a 'filter', so it automatically runs before any git operations, whether it's add, commit, diff or an interactive rebase.

[0] https://github.com/kynan/nbstripout

Re: Why we do machine learning engineering with YAML, not notebooks

#12
post #9
post #7

Earlier quoted context omitted.

Perhaps for young people it's actually necessary to mention JSON and YAML as you people tend to read the news rather than history or best practises textbooks?

Many hyped development and administration practices make me snarky, too, but let's not take it out on random (young) people on the Internet. There's a great deal of history and best practices that are being ignored not just by ADHD juniors, but also by team leads and managers of every seniority level. Lots of people out there spend decades in the industry, build solid careers based on playing the right office politic…

Ha! The parent might be right about the young people though I'd think the reason for reinventing things is a lot more people writing code and preferring different styles. That is a positive!

I don't think YAML/JSON are being superseded, but I'd really love for something like Cue (https://cuelang.org/) to become the standard for storing configuration.

Re: Why we do machine learning engineering with YAML, not notebooks

#13
post #9
post #7

Earlier quoted context omitted.

Perhaps for young people it's actually necessary to mention JSON and YAML as you people tend to read the news rather than history or best practises textbooks?

Many hyped development and administration practices make me snarky, too, but let's not take it out on random (young) people on the Internet. There's a great deal of history and best practices that are being ignored not just by ADHD juniors, but also by team leads and managers of every seniority level. Lots of people out there spend decades in the industry, build solid careers based on playing the right office politic…

> is YAML being "superseded"? By what now??)

i hope by something that isn't the kitchen sink... i actually prefer XML and i hate XML.

started looking at https://dhall-lang.org/# which compiles to json/yaml, is seriously strongly typed and explicitly not Turing-complete both as a design goal and current reality.

Re: Why we do machine learning engineering with YAML, not notebooks

#14
post #9
post #7

Earlier quoted context omitted.

Perhaps for young people it's actually necessary to mention JSON and YAML as you people tend to read the news rather than history or best practises textbooks?

Many hyped development and administration practices make me snarky, too, but let's not take it out on random (young) people on the Internet. There's a great deal of history and best practices that are being ignored not just by ADHD juniors, but also by team leads and managers of every seniority level. Lots of people out there spend decades in the industry, build solid careers based on playing the right office politic…

I read "young people" as "young in their career people". It's truly exasperating to see the wheel reinvented again, and again, and again, and again (I felt like typing that a lot more but I'll stop there :)). There are so many subtle lessions in this field, will there ever be any hope of recording and communicating them? Seems not likely.

Re: Why we do machine learning engineering with YAML, not notebooks

#16
So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.

Re: Why we do machine learning engineering with YAML, not notebooks

#17

So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.

Me too! I find notebooks are only good for teaching. No matter how hard I try, they never help me when doing my own analysis!

Re: Why we do machine learning engineering with YAML, not notebooks

#18

So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.

Netflix apparently. They've built an entire software framework for enabling their data scientists to put their notebooks into production [0].

[0] https://netflixtechblog.com/notebook-innovation-591ee3221233

Re: Why we do machine learning engineering with YAML, not notebooks

#19
post #6
post #5

The title is seriously misleading. They aren't doing their ML engineering in yaml. If you look at the snipper the article, you can see that their code is in flat .py files. The config is in yaml (which is also how everyone else uses it). It's like someone saying that they do their ML in a dockerfile.

The linked site (towardsdatascience) is kind of like medium blogs. High variability in quality with a lot of self-promoters and the occasional diamond in the rough. But a ton of rough.

It is a Medium blog. Its literally a Medium page, they just rely on other people's contributions which they have editors look over and then aggregate rather than produce their own. I suppose the editing process may help with quality some, but it really shouldn't be seen as much more than a typical blog.

Re: Why we do machine learning engineering with YAML, not notebooks

#20

For .ipynb notebooks, I highly recommend using nbstripout [0] to strip the Jupiter output before committing the notebooks to the repository (thus making the diffs sane). You can also set it up as a 'filter', so it automatically runs before any git operations, whether it's add, commit, diff or an interactive rebase. [0] https://github.com/kynan/nbstripout

Also a plug for https://nbdev.fast.ai/ which lets you run tests, produce documentation and version control notebooks.
Post reply on HN