Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What does your ML pipeline look like?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Ask HN: What does your ML pipeline look like?

#31

Post by lostdog »

Here are some things that have helped me make shipping models easier.

- Tag and version your datasets, especially your test set, so that you are confident when you compare performance across models or over time.

- Test your training pipeline. Run it on a single batch or a tiny dataset. The whole test should take less than 1 minute to confirm that your training pipeline didn't break. Once one test works, people will write others.

- You should be able to measure the accuracy of your model with a single command, or as part of your CI process. This command should spit out a single report (PDF or html) which is all you need to look at to decide whether to ship a new model or not, including run-time performance (if needed).

- Don't create hermetic training environments that prevent you from doing debugging. Sometimes you just need to ssh in and put in some print statements to track down a problem.

Re: Ask HN: What does your ML pipeline look like?

#32

Post by mistrial9 »

tomasdpinho wrote: post #9

As a DevOps Engineer working for a ML-based company and have had worked for others in the past, these are my quick suggestions for production readiness. DOs: If you are doing any kind of soft-realtime (i.e. not batch processing) inference, by exposing a model on a request-response lifecycle, use Tensorflow Serving for concurrency reasons. Version your models and track their training. Use something like MLFlow for tha…

Python 2.7 in 2019 is the best Python 2.7 there has ever been -- which is to say, it works very, very well. "more heat than light" on this particular topic. Do not start new projects in Python 2.7 -- ok fine. However, not done with Python 2.7 here

Re: Ask HN: What does your ML pipeline look like?

#33

Post by tixocloud »

jcims wrote: post #6

Would be interested if anyone in here has a pipeline operating on regulated data (HIPAA, financial, etc). Having a hard time drawing boundaries around what the data science team has access to for development and experimentation vs. where production pipelines would operate. (e.g. where in the process do people/processes get access to live data)

We're in the financial services space. The data science team has access to a data warehouse that gets updated daily but no access to production data directly so we use this for development and experimentation.

The definition of production environment varies for us. Production could mean running scheduled batch jobs on a separate 'production' pipeline from the data warehouse and results can stored in a database. Naturally, all our data is in-house at the moment so we have had to set everything up ourselves.

We're working on an enterprise-wide deployment system that integrates with the build/testing infrastructure where all the data science teams across the financial institution can leverage for deploying models and monitoring performance.

Re: Ask HN: What does your ML pipeline look like?

#34

Post by tixocloud »

We're building a deployment system based on the following principles that we've learned:

- Version control of models and datasets

- Code, naming conventions and formatting consistency

- Testing of models before deployment into production (in most cases, this helps gain credibility across engineering)

- Model review process with a senior data scientist

- Kubernetes/Docker for deployment serving as an API or running a scheduled job

- Model performance monitoring when in production to identify degradation

Re: Ask HN: What does your ML pipeline look like?

#35

Post by tixocloud »

fcolas wrote: post #21
tomasdpinho wrote: post #9

As a DevOps Engineer working for a ML-based company and have had worked for others in the past, these are my quick suggestions for production readiness. DOs: If you are doing any kind of soft-realtime (i.e. not batch processing) inference, by exposing a model on a request-response lifecycle, use Tensorflow Serving for concurrency reasons. Version your models and track their training. Use something like MLFlow for tha…

+1, tomasdpinho. Yes to everything, and notably the queues everywhere, versioning the models, and the issue to mix sync and async (go for queues). As a scientist designing risk management systems, I also like to: . avoid moving the data; . bring the (ML/stats) code to the data; . make in-memory computations (when possible) to reduce latency (network+disk); . work on live data instead of copies that drift out-of-date;…

+2 for bringing the (ML/stats) code to the data instead of the other way around

Re: Ask HN: What does your ML pipeline look like?

#36

Post by ethbro »

maksimum wrote: post #25
tomasdpinho wrote: post #13

Earlier quoted context omitted.

Containers are meant to be stateless infrastructure. By downloading something at startup, you're breaking that contract implicitly. Secondly, depending on where you're deploying, downloads from S3 (and then loading to memory) may take a non-negligible amount of time that can impact the availability of your pods (again, depending on their configuration). Synchronicity everywhere may cause request loss if your ML pipel…

> Containers are meant to be stateless infrastructure. By downloading something at startup, you're breaking that contract implicitly. I feel that mounting a NFS partition is a similar break of contract. I.e. you could see the same image behave differently depending on what's in the NFS partition. I feel like to get data in a "reproducible" way you need to pull it from a data versioning system. I think there's differe…

Curious about how you'd scale with data versioning.

In any type of realtime, high bandwidth feed, I feel like what you're suggesting isn't cost effective for the benefits it provides.

If you need absolute reproducibility and back-testing or your feed is lower bandwidth, it maybe makes sense. But not for larger systems.

Re: Ask HN: What does your ML pipeline look like?

#37

Post by shishy »

reureu wrote: post #17
jcims wrote: post #6

Would be interested if anyone in here has a pipeline operating on regulated data (HIPAA, financial, etc). Having a hard time drawing boundaries around what the data science team has access to for development and experimentation vs. where production pipelines would operate. (e.g. where in the process do people/processes get access to live data)

I do machine learning work in healthcare, and work for a HIPAA covered entity. The issue of permissions and data access often gets applied in an unnecessarily strict fashion to data scientists in these environments, often due to a lack of understanding from engineering managers who have been brought in from a non-regulated environment (e.g., hiring a salesforce engineering manager into a healthcare system so they can…

Not the OP, but I'd love to chat more about your experiences in this space. I don't see an email in your profile; feel free to drop me a note using the one in my email.

Re: Ask HN: What does your ML pipeline look like?

#38

Post by SkyPuncher »

reureu wrote: post #17
jcims wrote: post #6

Would be interested if anyone in here has a pipeline operating on regulated data (HIPAA, financial, etc). Having a hard time drawing boundaries around what the data science team has access to for development and experimentation vs. where production pipelines would operate. (e.g. where in the process do people/processes get access to live data)

I do machine learning work in healthcare, and work for a HIPAA covered entity. The issue of permissions and data access often gets applied in an unnecessarily strict fashion to data scientists in these environments, often due to a lack of understanding from engineering managers who have been brought in from a non-regulated environment (e.g., hiring a salesforce engineering manager into a healthcare system so they can…

I'd just like to point out that "the minimum necessary for their job" is the reason many engineering managers apply unnecessarily strict rules.

It's very difficult to build rules and policies that allow broad access while maintain minimum necessary. Some project may be completely justified in accessing "all" (waves hands) data at its conception but slowly morphing to focus on only a few key identifiers while still processing "all" data.

Re: Ask HN: What does your ML pipeline look like?

#40

Post by maksimum »

ethbro wrote: post #36
maksimum wrote: post #25

Earlier quoted context omitted.

> Containers are meant to be stateless infrastructure. By downloading something at startup, you're breaking that contract implicitly. I feel that mounting a NFS partition is a similar break of contract. I.e. you could see the same image behave differently depending on what's in the NFS partition. I feel like to get data in a "reproducible" way you need to pull it from a data versioning system. I think there's differe…

Curious about how you'd scale with data versioning. In any type of realtime, high bandwidth feed, I feel like what you're suggesting isn't cost effective for the benefits it provides. If you need absolute reproducibility and back-testing or your feed is lower bandwidth, it maybe makes sense. But not for larger systems.

Interesting topic. :)

This is mainly relevant if your data is used for training.

It seems like you'd want to use a log-based system like kafka to manage versioning and state in this case. I imagine you could:

1. Store incoming training data in a "raw data" topic.

2. A model trainer consumes incoming training data, updates a model's state, and at a pre-determined period writes the model's state as of a given offset in the "raw data" topic in a "model state checkpoint" topic.

3. Then you probably have some "regression testing" workflow that reads from the "model state checkpoint" topic and upon success writes to a "latest best model" topic.

4. Workers that use the model in production read from the "latest best model" topic and update their state upon a change.

I imagine you could add constraints about "model" continuity or gradual release to production that would make the process more complex, but I feel like fundamentally kafka solves a lot of the distributed systems problems.

Post reply on HN