Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
1–10 of 40 posts
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#2Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI (https://github.com/iterative/cml).
CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML compatible with CI:
1. In ML, pass/fail tests aren’t enough. Understanding model performance might require data visualizations and detailed metric reports. CML automatically generates custom reports after every CI run with visual elements like tables and graphs. You can even get a Tensorboard.dev link as part of your report.
2. Dataset changes need to trigger feedback just like source code. CML works with DVC so dataset changes trigger automatic training and testing.
3.Hardware for ML is an ecosystem in itself. We’ve developed use cases with CML and Docker Machine to automatically provision and deploy cloud compute instances (CPU & GPU) for model training.
Our philosophy is that ML projects- and MLOps practices- should be built on top of traditional software tools and CI systems, and not as a separate platform. Our goal is to extend DevOps’ wins from software development to ML. Check out our project site (https://cml.dev) and repo, and please let us know what you think!
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#3Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI ( https://github.com/iterative/cml ). CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML comp…
This is really cool. We've been recommending DVC to our users for a long time, and this looks like a natural step forward for the Iterative ecosystem.
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#4Very interesting, I've been looking for something like this to add to our ML pipeline. a few questions:
1) can we see examples of generated reports?
2) what happens if training fails?
3) what kind of metrics can it graph? can we have it track our custom metrics?
4) can we connect with external services like with webhooks,slack, and other integrations.
5) is this a docker technology, or how does it deal with images and dependencies?
Great work!
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#5Very interesting, I've been looking for something like this to add to our ML pipeline. a few questions: 1) can we see examples of generated reports? 2) what happens if training fails? 3) what kind of metrics can it graph? can we have it track our custom metrics? 4) can we connect with external services like with webhooks,slack, and other integrations. 5) is this a docker technology, or how does it deal with images an…
Thanks, and good questions!
1. Yes! Let me link some reports and example repos:
- A basic classification problem with scikit learn: https://github.com/iterative/cml_base_case/pull/2
- CML with DVC & Vega-Lite graphs: https://github.com/iterative/cml_dvc_case/pull/4
- Neural style transfer with EC2 GPU: https://github.com/iterative/cml_cloud_case/pull/2
2. If training fails, you'll be notified that your run failed in the GitHub Action dashboard (or GitLab CI/CD dashboard). See here for some real life examples of failure ;) : https://github.com/iterative/cml_cloud_case/actions
3. CML reports are markdown documents, so you can write any kind of text to them. If your metrics are output in a file `metrics.txt`, you can have your runner execute `cat metrics.txt >> report.md` and then have CML pass on the report to GitHub/GitLab. Likewise, any graphing library is supported because you can add standard image files (.png, .jpg) to the report. So custom metrics and custom graphs. We like DVC for managing and plotting metrics, but we're biased because we also maintain it.
4. Yep, GitHub Actions is pretty powerful and flexible. Works with whatever external services you can connect to your Action!
5. It's not strictly a Docker technology. We use Docker images preinstalled with the CML library in our examples, but you can just install the library with npm in your own image. https://github.com/iterative/cml#using-your-own-docker-image
Let me know if there's anything else I can tell you about
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#6ML is a relatively young field, and decades behind Software Engineering in terms of best practices for running production systems. CI/CD massively improved the innovation cycle time and quality of production software, and I believe it is key for building robust production ML systems as well. CML looks like a really easy to use product for bringing CI/CD to ML projects.
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#7Awesome to see a github native workflow for CI/CD in the ML space! This team is closet I seen that's like Hashicorp for ML
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#8Does it has any hidden fees or is totally open source?
Can I have GPUs on premise?
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#9Awesome to see a github native workflow for CI/CD in the ML space! This team is closet I seen that's like Hashicorp for ML
Thanks! Hashicorp is our inspiration :D
Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects
#10Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI ( https://github.com/iterative/cml ). CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML comp…
This is really cool. We've been recommending DVC to our users for a long time, and this looks like a natural step forward for the Iterative ecosystem.
Yeah, we were seeing a lot of users organically asking about CI/CD. DVC had to come first to address some of the data management issues, which seemed like the biggest hurdle to CI. But we're excited to add this now.