You can also set it up as a 'filter', so it automatically runs before any git operations, whether it's add, commit, diff or an interactive rebase.
Why we do machine learning engineering with YAML, not notebooks
11–20 of 31 posts
Re: Why we do machine learning engineering with YAML, not notebooks
#12Earlier quoted context omitted.
Perhaps for young people it's actually necessary to mention JSON and YAML as you people tend to read the news rather than history or best practises textbooks?
Many hyped development and administration practices make me snarky, too, but let's not take it out on random (young) people on the Internet. There's a great deal of history and best practices that are being ignored not just by ADHD juniors, but also by team leads and managers of every seniority level. Lots of people out there spend decades in the industry, build solid careers based on playing the right office politic…
I don't think YAML/JSON are being superseded, but I'd really love for something like Cue (https://cuelang.org/) to become the standard for storing configuration.
Re: Why we do machine learning engineering with YAML, not notebooks
#13Earlier quoted context omitted.
Perhaps for young people it's actually necessary to mention JSON and YAML as you people tend to read the news rather than history or best practises textbooks?
Many hyped development and administration practices make me snarky, too, but let's not take it out on random (young) people on the Internet. There's a great deal of history and best practices that are being ignored not just by ADHD juniors, but also by team leads and managers of every seniority level. Lots of people out there spend decades in the industry, build solid careers based on playing the right office politic…
i hope by something that isn't the kitchen sink... i actually prefer XML and i hate XML.
started looking at https://dhall-lang.org/# which compiles to json/yaml, is seriously strongly typed and explicitly not Turing-complete both as a design goal and current reality.
Re: Why we do machine learning engineering with YAML, not notebooks
#14Earlier quoted context omitted.
Perhaps for young people it's actually necessary to mention JSON and YAML as you people tend to read the news rather than history or best practises textbooks?
Many hyped development and administration practices make me snarky, too, but let's not take it out on random (young) people on the Internet. There's a great deal of history and best practices that are being ignored not just by ADHD juniors, but also by team leads and managers of every seniority level. Lots of people out there spend decades in the industry, build solid careers based on playing the right office politic…
Re: Why we do machine learning engineering with YAML, not notebooks
#15Re: Why we do machine learning engineering with YAML, not notebooks
#16Re: Why we do machine learning engineering with YAML, not notebooks
#17So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.
Re: Why we do machine learning engineering with YAML, not notebooks
#18So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.
[0] https://netflixtechblog.com/notebook-innovation-591ee3221233
Re: Why we do machine learning engineering with YAML, not notebooks
#19The title is seriously misleading. They aren't doing their ML engineering in yaml. If you look at the snipper the article, you can see that their code is in flat .py files. The config is in yaml (which is also how everyone else uses it). It's like someone saying that they do their ML in a dockerfile.
The linked site (towardsdatascience) is kind of like medium blogs. High variability in quality with a lot of self-promoters and the occasional diamond in the rough. But a ton of rough.
Re: Why we do machine learning engineering with YAML, not notebooks
#20For .ipynb notebooks, I highly recommend using nbstripout [0] to strip the Jupiter output before committing the notebooks to the repository (thus making the diffs sane). You can also set it up as a 'filter', so it automatically runs before any git operations, whether it's add, commit, diff or an interactive rebase. [0] https://github.com/kynan/nbstripout