Experienced machine learning professionals - How do you create scalable, deployable and reproducible data/ML pipelines at your work?
Ask HN: What does your ML pipeline look like?
1–10 of 50 posts
Re: Ask HN: What does your ML pipeline look like?
#2Depends on what you're trying to do.
Are you putting a trained inference model into production as a product? Is it a RL system (completely different architecture than an inference system)? Are you trying to build a model with your application data from scratch? Are you doing NLP or CV?
As a rule of thumb I look at the event diagram of the application/systems you're trying to implement ML into, which should tell you how to structure your data workflows in line with the existing data flows of the application. If it's strictly a constrained research effort then pipelines are less important, so go with what's fast and easy to document/read.
Generally speaking, you want your ML/DS/DE systems to be loosely coupled to your application data structures - with well defined RPC standards informed by the data team. I generally hate data pooling, but if we're talking about pooling 15 microservices vs pooling 15 monoliths, then the microservices pooling might be necessary.
Realistically this ends up being a business decision based on organizational complexity.
Re: Ask HN: What does your ML pipeline look like?
#3Obligatory XKCD: https://xkcd.com/2054/
Re: Ask HN: What does your ML pipeline look like?
#4Here’s a framework we’ve been developing for this purpose, delivered as a python cookiecutter:
https://github.com/hackalog/cookiecutter-easydata
The framework makes use of conda environments, python 3.6+, makefiles, and jupyter notebooks, so if that tooling fits into your data/ML workflow.
We gave a tutorial on the framework at Pydata NYC, and it’s still very much in active development - we refine it with every new project we work on. The tutorial is available from:
https://github.com/hackalog/bus_number
While it works well for our projects, the more real-world projects we throw at it, the better the tooling gets, so we’d love to hear from anyone who want to give it a shot.
Re: Ask HN: What does your ML pipeline look like?
#5At Logical Clocks, we build a horizontally scalable ML pipeline framework on the open-source Hopsworks platform, based around its feature store and Airflow for orchestration:
* https://hopsworks.readthedocs.io/en/latest/hopsml/hopsML.htm...
* https://www.logicalclocks.com/feature-store/
The choice for the DataPrep stage basically comes down to Spark or Apache Beam, and we currently support Spark, but plan to soon add support for Beam, because of some of the goodies in TFX (TensorFlow Extended).
For distributed hyperparam opt and distributed training, we leverage Apache Spark and our own version of YARN that supports GPUs -
* https://www.youtube.com/watch?v=tx6HyoUYGL0
For model serving, we support Kubernetes:
* https://hopsworks.readthedocs.io/en/0.9/hopsml/hopsML.html#s...
Our platform supports TLS/SSL certificates everywhere and is open-source. Download it and try it, and it runs in several large enterprises in Europe. We have a cluster with >1000 users in Sweden here:
(Edited for formatting)
Re: Ask HN: What does your ML pipeline look like?
#6Would be interested if anyone in here has a pipeline operating on regulated data (HIPAA, financial, etc). Having a hard time drawing boundaries around what the data science team has access to for development and experimentation vs. where production pipelines would operate. (e.g. where in the process do people/processes get access to live data)
Re: Ask HN: What does your ML pipeline look like?
#7At Logical Clocks, we build a horizontally scalable ML pipeline framework on the open-source Hopsworks platform, based around its feature store and Airflow for orchestration: * https://hopsworks.readthedocs.io/en/latest/hopsml/hopsML.htm... * https://www.logicalclocks.com/feature-store/ The choice for the DataPrep stage basically comes down to Spark or Apache Beam, and we currently support Spark, but plan to soon add…
This is great, thanks for the link.
Could you expand on how this workflow be different/better than sticking to just something like TFX and tensorflow serving? Is it easier to use or more scalable?
Re: Ask HN: What does your ML pipeline look like?
#8Depends on what you're trying to do. Are you putting a trained inference model into production as a product? Is it a RL system (completely different architecture than an inference system)? Are you trying to build a model with your application data from scratch? Are you doing NLP or CV? As a rule of thumb I look at the event diagram of the application/systems you're trying to implement ML into, which should tell you h…
Thanks for the reply. Could you give some more insight into how and what tools you choose for the different sort of tasks (say NLP vs CV vs RL)? Also, how and why are different tools/pipelines better for production and product building?
Re: Ask HN: What does your ML pipeline look like?
#9As a DevOps Engineer working for a ML-based company and have had worked for others in the past, these are my quick suggestions for production readiness.
DOs:
If you are doing any kind of soft-realtime (i.e. not batch processing) inference, by exposing a model on a request-response lifecycle, use Tensorflow Serving for concurrency reasons.
Version your models and track their training. Use something like MLFlow for that. Divise a versioning system that makes sense for your organization.
If you are using Kubernetes in Production, mount NFS in your containers to serve models. Do not download anything (from S3, for instance) on container start up time unless your models are small (If you have to write some sort of heavy preprocessing or postprocessing steps, eventually port them to a more efficient language than Python. Say Go, Rust, etc.
DO NOTs:
Do NOT make your ML engineers/researchers write anything above the model stack. Don't make them write queue management logic, webservers, etc. That's not their skillset, they will write poorer and less performant code. Bring in a Backend Engineer EARLY.
Do NOT mix and match if you are working on an asynchronous model, i.e. don't have a callback-based API and then have a mix of queues and synchronous HTTP calls. Use queues EVERYWHERE.
DO NOT start new projects in Python 2.7. From past experiences, some ML engineers/researchers are quite attached to the older versions of Python. These are ending support in 2020 and it makes no sense to start a project using them now.
Re: Ask HN: What does your ML pipeline look like?
#10At Logical Clocks, we build a horizontally scalable ML pipeline framework on the open-source Hopsworks platform, based around its feature store and Airflow for orchestration: * https://hopsworks.readthedocs.io/en/latest/hopsml/hopsML.htm... * https://www.logicalclocks.com/feature-store/ The choice for the DataPrep stage basically comes down to Spark or Apache Beam, and we currently support Spark, but plan to soon add…
This is great, thanks for the link. Could you expand on how this workflow be different/better than sticking to just something like TFX and tensorflow serving? Is it easier to use or more scalable?
It is pretty much the same as TFX - but with Spark for both DataPrep and Distributed HyperparamOpt/Training, and a Feature Store. Model serving is slightly more sophisticated than just TensorFlow Serving on Kubernetes. We support serving requests through the Hopsworks REST API to TFServering/Kubernetes. This gives us both access control (clients have a TLS cert to authenticate and authorize themselves) and we log all predictions to a Kafka topic. We are adding support to enrich feature vectors using the Feature Store in the serving API, not quite there yet.
We intend to support TFX as we already support Flink. Flink/Beam for Python 3 is needed for TFX, but it's not quite there yet, almost.
It will be interesting to see which one of Spark or Beam will become the horizontally scalable platform of choice for TensorFlow. (PyTorch people don't seem as interested, as they mostly come from a background of not wanting complexity).