Ask HN: What is your ML stack like?

#1

Post by imagiko » Tue, Nov 12, 2019, 5:58 PM UTC

How did your team build out AI/ML pipelines and integrated it with your existing codebase? For example, how did your backend team(using Java?) work in sync with data teams (using R or python?) to have minimal rewriting/glue code as possible to deploy models in production. What were your architectural decisions that worked, or didn't?

I'm currently working to make an ML model written in R work on our backend system written in Java. After the dust settles I'll be looking for ways to streamline this process.

Re: Ask HN: What is your ML stack like?

#2

Post by marcinzm » Tue, Nov 12, 2019, 6:41 PM UTC

The systems I've seen basically break things into different services. Tied together with gRPC or Thrift which have code generators for most languages. So the Java backend simply makes RPC requests to a server running R.

Although in one case we had very tight latency requirements (ie: 10ms) so the ML results were pre-computed and loaded from a cache on the backend servers.

Re: Ask HN: What is your ML stack like?

#3

Post by gautamcgoel » Tue, Nov 12, 2019, 6:42 PM UTC

I'm a PhD student at Caltech, working on the theoretical foundations of ML. I personally don't do a lot of coding, but basically everyone in my department uses Python (especially Pytorch) for deep learning/ML. This all runs on Nvidia GPUs (never seen an AMD GPU in the office). Occasionally people code in Matlab, especially if they work in optimization or control. Tmux and git are the only command line tools I see commonly used. Occasionally people ssh into an Amazon box if they need more compute.

Re: Ask HN: What is your ML stack like?

#4

Post by sam0x17 » Tue, Nov 12, 2019, 6:46 PM UTC

crystal / shainet (https://github.com/NeuraLegion/shainet)

I contract for some clients in fintech and some defense-related stuff.

Re: Ask HN: What is your ML stack like?

#5

Post by aaron-santos » Tue, Nov 12, 2019, 6:47 PM UTC

What didn't work:

Shipping pickled models to other teams.

Deploying Sagemaker endpoints (too costly).

Requiring editing of config files to deploy endpoints.

What did work:

Shipping http endpoints.

Deriving api documentation from model docstrings.

Deploying lambdas (less costly than Sagemaker endpoints).

Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/output data.

Continuous deployment (after model is saved no manual intervention if model response matches output data).

Re: Ask HN: What is your ML stack like?

#6

Post by exhilaration » Tue, Nov 12, 2019, 6:49 PM UTC

aaron-santos wrote: post #5

What didn't work: Shipping pickled models to other teams. Deploying Sagemaker endpoints (too costly). Requiring editing of config files to deploy endpoints. What did work: Shipping http endpoints. Deriving api documentation from model docstrings. Deploying lambdas (less costly than Sagemaker endpoints). Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/o…

Could you possibly define "pickling" in this context for us ML noobs?

Re: Ask HN: What is your ML stack like?

#7

Post by rococode » Tue, Nov 12, 2019, 6:52 PM UTC

exhilaration wrote: post #6

aaron-santos wrote: post #5

What didn't work: Shipping pickled models to other teams. Deploying Sagemaker endpoints (too costly). Requiring editing of config files to deploy endpoints. What did work: Shipping http endpoints. Deriving api documentation from model docstrings. Deploying lambdas (less costly than Sagemaker endpoints). Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/o…

Could you possibly define "pickling" in this context for us ML noobs?

There's a serialization module called pickle that can be used to store models:

https://docs.python.org/3/library/pickle.html

Re: Ask HN: What is your ML stack like?

#8

Post by barbecue_sauce » Tue, Nov 12, 2019, 6:53 PM UTC

exhilaration wrote: post #6

aaron-santos wrote: post #5

What didn't work: Shipping pickled models to other teams. Deploying Sagemaker endpoints (too costly). Requiring editing of config files to deploy endpoints. What did work: Shipping http endpoints. Deriving api documentation from model docstrings. Deploying lambdas (less costly than Sagemaker endpoints). Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/o…

Could you possibly define "pickling" in this context for us ML noobs?

"Pickling" is just the pythonic term for serialization. In this context, it most likely means persisting the model to disk as some sort of file.

Re: Ask HN: What is your ML stack like?

#9

Post by [deleted] » Tue, Nov 12, 2019, 6:54 PM UTC

[deleted]

Re: Ask HN: What is your ML stack like?

#10

Post by teebeck » Tue, Nov 12, 2019, 6:56 PM UTC

exhilaration wrote: post #6

aaron-santos wrote: post #5

What didn't work: Shipping pickled models to other teams. Deploying Sagemaker endpoints (too costly). Requiring editing of config files to deploy endpoints. What did work: Shipping http endpoints. Deriving api documentation from model docstrings. Deploying lambdas (less costly than Sagemaker endpoints). Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/o…

Could you possibly define "pickling" in this context for us ML noobs?

Pickling isn't ML-specific. Pickle is an object serialization library in Python.