21–30 of 31 posts
The title is seriously misleading. They aren't doing their ML engineering in yaml. If you look at the snipper the article, you can see that their code is in flat .py files. The config is in yaml (which is also how everyone else uses it). It's like someone saying that they do their ML in a dockerfile.
So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.
So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.
Have you never seen a script in production?
The topic of production notebooks often shows up. I’ve seen tools like papermill and dagster being used for notebook prod, just like in Netflix.
I concluded that using notebooks for prod is always a tech decision, often influence by a tradeoff: risk for premature optimization (writing scripts early on in the project that may only be used once) and underengineering (using non-maintainable and clunky code to support mission-critical workloads): https://ljvmiranda921.github.io/notebook/2020/03/16/jupyter-...
So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.
Netflix apparently. They've built an entire software framework for enabling their data scientists to put their notebooks into production [0]. [0] https://netflixtechblog.com/notebook-innovation-591ee3221233
We don't deploy or execute ML models in production as notebooks. We have many other solutions for that use case. In particular, check out https://metaflow.org
For .ipynb notebooks, I highly recommend using nbstripout [0] to strip the Jupiter output before committing the notebooks to the repository (thus making the diffs sane). You can also set it up as a 'filter', so it automatically runs before any git operations, whether it's add, commit, diff or an interactive rebase. [0] https://github.com/kynan/nbstripout
So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.
Papermill (https://papermill.readthedocs.io/en/latest/) made it extremely easy, to the point where I question the real value of moving away from this model. But software engineers practically hiss when you mention Jupyter because it's too different from the rest of their tooling.
Reasons 2 and 3 for avoiding jupyter are more justified but easy enough to work around between jupytext and papermill.
So, every month or two I see another article tut-tutting people for putting notebooks into production, and I'm curious, who is actually doing this? I've never seen such a thing in the wild, and I'm genuinely (morbidly?) curious what it would look like in practice.
I really dislike this practice. Code that runs in production should be code reviewed and there should be some monitoring in place to make sure the job is working correctly
Its quite simple to develop a model, bundle it up for deployment and deploy. Nobody cares about your fancy YAML based containerized deployment and monitoring setup, everyone has that. The challenge comes in when you have a continuous cycle of data ingestion to model optimization, training, evaluation and deployment. Pretty much everybody has huge amount of code duplication in there. It also comes from the fact that m…