Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cortex: Deploy machine learning models in production

github.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Cortex: Deploy machine learning models in production

#11

Post by deliahu »

mzanchi wrote: post #8

Calling it an alternative to SageMaker might be a bit misleading, as SageMaker is also a platform for training the models in automatically allocated EC2 resources, even on spot instances.

Cortex contributor here - you're right, I would say we can be compared to SageMaker model deployment. We are currently working on supporting spot instances for serving, and training is on our roadmap.

Re: Cortex: Deploy machine learning models in production

#16

Post by ospillinger »

kuu wrote: post #15

How does this work under the hood? Is the model loaded every time it receives a request? Is it run in a docker or a lambda? How does it work after "uploading it" to amazon?

Each model is loaded into a Docker container, along with any Python packages and request handling code. The cluster runs on EKS on your AWS account. Cortex takes the declarative configuration from 'cortex.yaml' and creates it every time you run 'cortex deploy' so the containers don’t change unless you run 'cortex deploy' again with updated configuration. This post goes into more detail about some of our design decisions: https://towardsdatascience.com/inference-at-scale-49bc222b3a...

Re: Cortex: Deploy machine learning models in production

#17

Post by kuu »

ospillinger wrote: post #16
kuu wrote: post #15

How does this work under the hood? Is the model loaded every time it receives a request? Is it run in a docker or a lambda? How does it work after "uploading it" to amazon?

Each model is loaded into a Docker container, along with any Python packages and request handling code. The cluster runs on EKS on your AWS account. Cortex takes the declarative configuration from 'cortex.yaml' and creates it every time you run 'cortex deploy' so the containers don’t change unless you run 'cortex deploy' again with updated configuration. This post goes into more detail about some of our design decisi…

Thank you!

Re: Cortex: Deploy machine learning models in production

#18

Post by oli5679 »

If your model can be exported as PMML, this is really nice. Fast, minimalist, battle-tested and with very clean API.

When I've tested, it's up to 10x faster than Flask + serialised model object and uses far less CPU resources.

Plays nicely with lightgbm and Xgboost.

https://github.com/openscoring/openscoring

Re: Cortex: Deploy machine learning models in production

#19

Post by solidasparagus »

ospillinger wrote: post #14
punnerud wrote: post #13

Seems to work with: Pytorch, TensorFlow, Keras, XGBoost, sklearn

Yes, Cortex uses ONNX Runtime ( https://github.com/microsoft/onnxruntime ) under the hood so any model that can be exported to ONNX can be deployed.

Is it only able to handle ONNX models? That's a pretty massive limitation compared to a hosted SageMaker endpoint.
Post reply on HN