We used TensorFlow Serving running in a docker container that is a rest endpoint for predictions. Then the backend can be agnostic to Serving. We personally used Node to query the container for whatever we needed.
Ask HN: What is your ML stack like?
101–110 of 134 posts
Re: Ask HN: What is your ML stack like?
#102> or example, how did your backend team(using Java?) work in sync with data teams
I guess we are a bit of an outlier, but we deploy the ML using Java / JVM. Not really in the same league as others here so the models are simple enough that the various Java ML frameworks are fine for it (DL4J, Smile, etc). We even do a lot of the interactive exploratory / training type work on the JVM (though via Groovy and Scala with BeakerX notebooks [1] - sometimes combined with Python and R).
I think as the field matures a lot more could move to this model.
Re: Ask HN: What is your ML stack like?
#103> or example, how did your backend team(using Java?) work in sync with data teams I guess we are a bit of an outlier, but we deploy the ML using Java / JVM. Not really in the same league as others here so the models are simple enough that the various Java ML frameworks are fine for it (DL4J, Smile, etc). We even do a lot of the interactive exploratory / training type work on the JVM (though via Groovy and Scala with…
We once had a team member re-write their R code in Java, using weka just to avoid too much hassle for the back-end team. So I guess you're not alone!
Re: Ask HN: What is your ML stack like?
#104our team built a Postgres backend on a single physical server (with a lot of good disk in RAID and others), python ran against the database, calling sklearn libs .. (skipping problem specific libs involved, but they added to either PG or the python side) Worked really, really well, easy to work on .. good architectural separation of stages in the process. Completed and shipped to an impressed customer. No GPUs
Re: Ask HN: What is your ML stack like?
#105For our current projects we use our open source Apache Spark framework Arc (https://arc.tripl.ai/) for feature prep then depending on the type of model we will either:
- use builtin Spark ML models
- call a model running as a service
- write files for a model to ingest (for a legacy project)
- develop a custom plugin or UDF (for calling via SQL)
We have built in stages for running Spark ML models in the framework as well as HTTP and Tensorflow Serving stages to call services. We recently ran a series of models for NLP that were in Python and Ocaml via the HTTP stage sending payload either in JSON or other formats that the services needed. The text extraction via OCR (tesseract) had been done as a prior Spark stage. This design allows us to call these more custom ML models but keep them part of a larger Spark job and use SQL and other features when needed. The services where deployed in AWS Fargate to allow for scaling. For other jobs we are deploying our Arc jobs using Argo for orchestration. We spin up compute on demand vs running inside a persistent cluster.
For training we use Jupyter Notebooks where possible. We have a plugin that generates Arc jobs from these notebooks.
For special cases we can add custom plugins or UDF functions to extend the framework. I have done similar plugins to run XGBoost models in Spark for example.
Whilst we try to be prescriptive around the ML stack for Data Scientists this approach has allow flexibility where needed and for different teams to own their part of the job. This is particularly useful in larger teams where development is more federated.
Re: Ask HN: What is your ML stack like?
#106We use our own product https://PI.EXCHANGE
We deploy the models as HTTP endpoints and consume them in R/Python/Excel.
We also have advanced functionality available to enterprise clients that are exposed as APIs with a customised JSON format to trigger various agents.
Re: Ask HN: What is your ML stack like?
#107Earlier quoted context omitted.
Could you possibly define "pickling" in this context for us ML noobs?
"Pickling" is just the pythonic term for serialization. In this context, it most likely means persisting the model to disk as some sort of file.
Not really - "pickling" an object in python is applying a very specific serialization protocol. That protocol happens to be built into the python language itself, but there are alternatives.
Re: Ask HN: What is your ML stack like?
#108What has worked fairly well so far: Models: - Models are structured as python packages, each model inherits a base class - base class has define how to train, and how to predict (as well as a few other more specific things) - ML engineer can override model serialization methods, default is just pickle Infra: - Code is checked in to github, Docker container built each merge into master - Use Sagemaker BYO container to…
> The tradeoff has been theres a step between notebook and production for ML engineers which can slow them down, but it forces code review and increases the number of tests checked in. This was a game-changer for us. What does your testing story look like?
We're still ironing out a few things but unit tests for various functions then we have smol statistically representative datasets for each model. In CI we train a model on the small dataset (aim for <5 mins e2e) then have a a suite of model metrics we care about, tests confirm values are within acceptable bounds and the test values are pulled into the PR.
Re: Ask HN: What is your ML stack like?
#109Earlier quoted context omitted.
Could you possibly define "pickling" in this context for us ML noobs?
If by ML noob you mean to say that you're like me and have zero formal CS training (as in, I don't know what a data structure is), pickling lets you write your Python workspace to a file just like Matlab's .mat file loading. It's excellent for writing scripts defining different parts of a data pipeline, or just for debugging/trying new things without waiting 20 minutes for something to filter.
Here is a simple explanation:
1. The universe is composed of things.
2. We can use the computer to store information about those things (this is the data).
3. In order to gain useful insights about those things, we want to do operations on the data. I.e. to compute. computation is done by algorithms.
4. Data structures are the bridge between the data about things and the algorithm. They hold the data such that the algorithm will have an easier time computing.
Re: Ask HN: What is your ML stack like?
#110Earlier quoted context omitted.
What about the A/B testing? What do you use for A/B strategy. How many predictions are being served by the model per second?
For most of the stuff we’ve deployed, we’re not yet operating at a scale/level of interest where A/B rearing is worth it. Additionally, the purposes we’re using most of these models for don’t really necessitate A/B testing. When we do need A/B testing, we’ll probably use something like Seldon. As for predictions/second, not very much at the moment: 1 per 30 seconds maybe? It’s not deployed into a Kubernetes cluster b…
So how do you know if a new version of a model is better than the existing serving version?