Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

cml.dev

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#31

Post by rhythmvertigo »

gravypod wrote: post #30
rhythmvertigo wrote: post #2

Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI ( https://github.com/iterative/cml ). CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML comp…

This looks really amazing. Do you have any plans to support on-prem GPUs? If we had Gitlab runners with GPUs in them would this project pick it up and use those runners for training/analysis?

Yep, it will work with your GPUs! You just set up self-hosted runners as usual https://docs.gitlab.com/runner/

GitHub & GitLab have both made it quite easy to use your own resources as runners. I recently met someone who was doing Actions with a Jetson Nano on their dresser :)

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#32

Post by gravypod »

rhythmvertigo wrote: post #31
gravypod wrote: post #30

Earlier quoted context omitted.

This looks really amazing. Do you have any plans to support on-prem GPUs? If we had Gitlab runners with GPUs in them would this project pick it up and use those runners for training/analysis?

Yep, it will work with your GPUs! You just set up self-hosted runners as usual https://docs.gitlab.com/runner/ GitHub & GitLab have both made it quite easy to use your own resources as runners. I recently met someone who was doing Actions with a Jetson Nano on their dresser :)

That's really cool. I might have to play around with this. Do you have any docs on what you do to deploy a model? Something I've been doing at work is dealing with the output of some ML code we have. We end up with ~150GB of data that needs to be synced to a file share in prod. I'm assuming DVC can be used for this.

After the run, output files, upload as a data set in DVC or something?

Documenting this full workflow would save a lot of confused devopsy people (like myself) survive in the world of ML. Thanks for this hard work you've all put into this!

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#33

Post by dmpetrov »

gravypod wrote: post #32
rhythmvertigo wrote: post #31

Earlier quoted context omitted.

Yep, it will work with your GPUs! You just set up self-hosted runners as usual https://docs.gitlab.com/runner/ GitHub & GitLab have both made it quite easy to use your own resources as runners. I recently met someone who was doing Actions with a Jetson Nano on their dresser :)

That's really cool. I might have to play around with this. Do you have any docs on what you do to deploy a model? Something I've been doing at work is dealing with the output of some ML code we have. We end up with ~150GB of data that needs to be synced to a file share in prod. I'm assuming DVC can be used for this. After the run, output files, upload as a data set in DVC or something? Documenting this full workflow…

150Gb ML model file? That's cool!

Yes, DVC can help with that. Where the data lives? S3/GCS or just a server with SSH?

Disclaimer: I'm a creator of DVC.

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#34

Post by calebkaiser »

gravypod wrote: post #32
rhythmvertigo wrote: post #31

Earlier quoted context omitted.

Yep, it will work with your GPUs! You just set up self-hosted runners as usual https://docs.gitlab.com/runner/ GitHub & GitLab have both made it quite easy to use your own resources as runners. I recently met someone who was doing Actions with a Jetson Nano on their dresser :)

That's really cool. I might have to play around with this. Do you have any docs on what you do to deploy a model? Something I've been doing at work is dealing with the output of some ML code we have. We end up with ~150GB of data that needs to be synced to a file share in prod. I'm assuming DVC can be used for this. After the run, output files, upload as a data set in DVC or something? Documenting this full workflow…

Not to be too self-promotional here, but I'm a maintainer of Cortex, a model deployment platform that sounds like it might be useful: https://github.com/cortexlabs/cortex

With DVC/Cortex, you can set things up so that all you have to do is run `dvc push` to update your model and `cortex deploy` to deploy it.

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#35

Post by gravypod »

dmpetrov wrote: post #33
gravypod wrote: post #32

Earlier quoted context omitted.

That's really cool. I might have to play around with this. Do you have any docs on what you do to deploy a model? Something I've been doing at work is dealing with the output of some ML code we have. We end up with ~150GB of data that needs to be synced to a file share in prod. I'm assuming DVC can be used for this. After the run, output files, upload as a data set in DVC or something? Documenting this full workflow…

150Gb ML model file? That's cool! Yes, DVC can help with that. Where the data lives? S3/GCS or just a server with SSH? Disclaimer: I'm a creator of DVC.

They live in an Azure Storage Account (Azure's not-S3). They're mounted as a network storage into pods in our kube cluster. All we need to do to deploy is copy the data into `//` and the code finds the newest version and loads it up. So really I'm just looking for a way to abstract azure blob storage from my ML people and to allow them to do the equivalent of `docker tag...` to choose what we roll out.

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#36

Post by doppenhe »

rhythmvertigo wrote: post #2

Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI ( https://github.com/iterative/cml ). CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML comp…

Deployment, inference and management can participate in this as well!

Here is the missing part for a total e2e solution: https://github.com/marketplace/actions/algorithmia-ci-cd

{disclaimer, we built this Github action}

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#37

Post by ishcheklein »

gravypod wrote: post #35
dmpetrov wrote: post #33

Earlier quoted context omitted.

150Gb ML model file? That's cool! Yes, DVC can help with that. Where the data lives? S3/GCS or just a server with SSH? Disclaimer: I'm a creator of DVC.

They live in an Azure Storage Account (Azure's not-S3). They're mounted as a network storage into pods in our kube cluster. All we need to do to deploy is copy the data into `/ / ` and the code finds the newest version and loads it up. So really I'm just looking for a way to abstract azure blob storage from my ML people and to allow them to do the equivalent of `docker tag...` to choose what we roll out.

I think DVC (+CML) is a good solution for this. It "wraps" artifacts that you store into Git. And Git repo abstracts access to the cloud. In your case of the mounted storage it will look like `git pull` + `dvc checkout` after model is "merged" in the production/master branch.

CML can automate and make the process of preparing the model to be merged into that branch reliable, visible, robust, etc.

I'm happy to help with this flow, ping me on Twitter - @shcheklein in DM or ivan on DVC Discord.

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#38

Post by rhythmvertigo »

doppenhe wrote: post #36
rhythmvertigo wrote: post #2

Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI ( https://github.com/iterative/cml ). CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML comp…

Deployment, inference and management can participate in this as well! Here is the missing part for a total e2e solution: https://github.com/marketplace/actions/algorithmia-ci-cd {disclaimer, we built this Github action}

coooool! going to try this out :)

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#39

Post by davidortega »

doppenhe wrote: post #36
rhythmvertigo wrote: post #2

Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI ( https://github.com/iterative/cml ). CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML comp…

Deployment, inference and management can participate in this as well! Here is the missing part for a total e2e solution: https://github.com/marketplace/actions/algorithmia-ci-cd {disclaimer, we built this Github action}

Hi doppenhe, we have that part already implemented using cml-send-github-check and dvc metrics diff. You can compare the metric that you prefer with dvc and then just set the status of the github check uploading your full report. Of course, you can also fail the workflow as your Github action does, but I think is more useful to see it as a report in the check.

disclaimer: I'm work with CML

Re: Show HN: Continuous Machine Learning – CI/CD for Machine Learning Projects

#40

Post by davidortega »

gravypod wrote: post #30
rhythmvertigo wrote: post #2

Hi, I'm one of the project creators. Continuous Machine Learning (CML) is an open source project to help ML projects use CI/CD with Github Actions and Gitlab CI ( https://github.com/iterative/cml ). CML automatically generates human-readable reports with metrics and data viz in every pull/merge request, and helps you use storage and GPU/CPU resources from cloud services. CML addresses three hurdles for making ML comp…

This looks really amazing. Do you have any plans to support on-prem GPUs? If we had Gitlab runners with GPUs in them would this project pick it up and use those runners for training/analysis?

You can also deploy runners with GPU on premise using CML docker image with GPU already supported having to install only the ndivia drivers and nvidia-docker in your machine.

docker run --name myrunner -d -e RUNNER_IDLE_TIMEOUT=1800 -e RUNNER_LABELS=cml -e RUNNER_REPO=$my_repo_url -e repo_token=$my_repo_token dvcorg/cml-gpu-py3-cloud-runner

It works for Gitlab and Github. Just only point your url and repo token

Post reply on HN