We're currently running a single NVIDIA RTX2080 with Tensorflow 2.0 on a Windows 10 station. We'll soon be switching to a standard multi-GPU rig running an air gapped Linux distro. Linux seems overall much better for ML because of better Docker integration and tensor core support on the newer GPUs. Also, we'll probably be switching from Tensorflow to Pytorch for model development. Pytorch requires a little bit more code, but debugging is 10X easier.
Ask HN: What is your ML stack like?
21–30 of 134 posts
Re: Ask HN: What is your ML stack like?
#22The systems I've seen basically break things into different services. Tied together with gRPC or Thrift which have code generators for most languages. So the Java backend simply makes RPC requests to a server running R. Although in one case we had very tight latency requirements (ie: 10ms) so the ML results were pre-computed and loaded from a cache on the backend servers.
I imagine I will be breaking down into different services as well. An ML "blackbox" that makes a call to the back-end for data and returns a result/prediction. This could happen through an API. What kind of API to choose is still open.
I'm not very sure what you mean when you say the ML results were pre-computed?
Re: Ask HN: What is your ML stack like?
#23why not write the ML model in Java?
Re: Ask HN: What is your ML stack like?
#24What didn't work: Shipping pickled models to other teams. Deploying Sagemaker endpoints (too costly). Requiring editing of config files to deploy endpoints. What did work: Shipping http endpoints. Deriving api documentation from model docstrings. Deploying lambdas (less costly than Sagemaker endpoints). Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/o…
Could you possibly define "pickling" in this context for us ML noobs?
To add to the other responses, I would recommend Joblib (over pickle or cpickle)
Reasons here: https://stackoverflow.com/a/12617603/1868436
Re: Ask HN: What is your ML stack like?
#25We're currently running a single NVIDIA RTX2080 with Tensorflow 2.0 on a Windows 10 station. We'll soon be switching to a standard multi-GPU rig running an air gapped Linux distro. Linux seems overall much better for ML because of better Docker integration and tensor core support on the newer GPUs. Also, we'll probably be switching from Tensorflow to Pytorch for model development. Pytorch requires a little bit more c…
And how do you get data on the fly [prediction phase]. Do you have an API call you make to get data that your ML algorithm can munch on?
Re: Ask HN: What is your ML stack like?
#26We're currently running a single NVIDIA RTX2080 with Tensorflow 2.0 on a Windows 10 station. We'll soon be switching to a standard multi-GPU rig running an air gapped Linux distro. Linux seems overall much better for ML because of better Docker integration and tensor core support on the newer GPUs. Also, we'll probably be switching from Tensorflow to Pytorch for model development. Pytorch requires a little bit more c…
Why airgapped? Is it a business/security requirement? If you have to share the machine, does everybody have to thumb drive over their files to run with the big GPU?
Re: Ask HN: What is your ML stack like?
#27crystal / shainet ( https://github.com/NeuraLegion/shainet ) I contract for some clients in fintech and some defense-related stuff.
What kind of targets is crystal used on? Strictly x86_64 linux?
EDIT: Took a little digging but I found it [1] (and yes it's primarily x86_64 linux+macOS)
[1] https://github.com/crystal-lang/crystal/wiki/Platform-Suppor...
Re: Ask HN: What is your ML stack like?
#28Earlier quoted context omitted.
hi Aaron, We did exactly what works for you into a open source python library, github.com/bentoml/bentoml. It packages your model for you into a standardized format, that you can use it in multiply serving scenarios online serving with api endpoint, offline serving with spark udf, CLI access or import it as python module. It also helps you deploy to different platform such as lambda, sagemaker and others. Our value i…
It's been great seeing this space fill out with solutions in the last year. MLFlow[1] is another open source solution I have my eyes on. BentoML looks more cohesive than our homegrown solution because it targets a more general case. One of the things I would miss switching to BentoML would be automatic requirements generation. We use pipreqs[2] to generate a requirements.txt given a model instance. Any thoughts on th…
hi Aaron, I'm one of the BentoML aurthors - great suggestion on pipreqs, will look into incorparating that into BentoML!
It should be very straightforward adding support for saving/loading Statsmodels in BentoML. In fact you should also be able to just use the existing "PickleArtifact" in BentoML for statsmodel predictors too. We will add an example notebook for working with Statsmodels library soon!
Re: Ask HN: What is your ML stack like?
#29What didn't work: Shipping pickled models to other teams. Deploying Sagemaker endpoints (too costly). Requiring editing of config files to deploy endpoints. What did work: Shipping http endpoints. Deriving api documentation from model docstrings. Deploying lambdas (less costly than Sagemaker endpoints). Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/o…
Yes to pickling models!
Re: Ask HN: What is your ML stack like?
#30We have a bit of a problem like what you mention. Our backend/app is in Java but the DS/ML team generally works in python. The ML team basically doesn't ship production code.
Here are the artifacts we produce:
1. For new models we often build a demo endpoints/glue code written in python/flask that can be compared against the prod output in dev/psup.
2. Deep learning models (much of what I do personally): saved in TF saved model format. If it is an update to an existing model often it is just a drop-in replacement. If it is a brand new model i will often include a flask demo (the python code does proper data transformation before calling on tf). On production side, after testing/regression these model are deployed via tensorflow-serving containers and used as gRPC endpoint. For production, whatever data pre-processing needs to be done is written by the backend team, who compare preprocessing output with our demo.
3. Logistic regression/tree models: again, for new models we provide the demo but what goes into production are either csv (logistic regression) or json (tree) of the weights/decision boundaries which are used as resources by the backend team's Java code.
The overall flow is:
ETL (via apache airflow/custom code) => model training/feature engineering => (saved model file + flask demo endpoint/documentation on feature transformations) => dev incorporate model/test into java backend => comparison of demo vs java backend => regression of java backend (if they had previous versions of model) => psup (small amount of prod data duplicated and ran in parallel with prod) => prod (model deployed + monitored)
There is a caveat that we also do some batch processing/not really live analysis that is just done in python and then results are pushed wherever they need to be pushed. In this case we don't involve the backend/java team.