Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What does your ML pipeline look like?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Ask HN: What does your ML pipeline look like?

#21

Post by fcolas »

tomasdpinho wrote: post #9

As a DevOps Engineer working for a ML-based company and have had worked for others in the past, these are my quick suggestions for production readiness. DOs: If you are doing any kind of soft-realtime (i.e. not batch processing) inference, by exposing a model on a request-response lifecycle, use Tensorflow Serving for concurrency reasons. Version your models and track their training. Use something like MLFlow for tha…

+1, tomasdpinho. Yes to everything, and notably the queues everywhere, versioning the models, and the issue to mix sync and async (go for queues).

As a scientist designing risk management systems, I also like to:

. avoid moving the data;

. bring the (ML/stats) code to the data;

. make in-memory computations (when possible) to reduce latency (network+disk);

. work on live data instead of copies that drift out-of-date; and

. write software to keep models up to date because they drift with time too and that's a major, operationally un-noticed, and extremely costly problem.

I'm not yet into Tensor/ML-Flow, but I use R, JS, and Postgres, thereby relying on open-source eco-systems (and packages) that are:

. as standard as possible;

. well-maintained;

. with a long expected support; and

. as few dependencies as possible.

Re: Ask HN: What does your ML pipeline look like?

#22

Post by kamac »

tomasdpinho wrote: post #13
hadsed wrote: post #11

Earlier quoted context omitted.

Can you elaborate on why downloading from S3 at startup is a bad idea? And why not synchronous everywhere as opposed to always queues? Good points overall that I'd agree with.

Containers are meant to be stateless infrastructure. By downloading something at startup, you're breaking that contract implicitly. Secondly, depending on where you're deploying, downloads from S3 (and then loading to memory) may take a non-negligible amount of time that can impact the availability of your pods (again, depending on their configuration). Synchronicity everywhere may cause request loss if your ML pipel…

> By downloading something at startup, you're breaking that contract implicitly.

Nitpicking here, but if you can ensure that certain version is downloaded, then the contract isn't violated.

Re: Ask HN: What does your ML pipeline look like?

#23

Post by mallochio »

Liveanimalcams wrote: post #19

I'm not a professional but I built a pipeline for Makers Part List - It involves ingesting a video URL, converting the video into images, then storing the images in google storage. Once stored I trigger the model to classify the image. The images are then displayed to annotators who verify/relabel the images. Once I get enough new images the system creates a .csv and uploads it to googles autoML where it retrains my…

Interesting. Could you maybe expand on the tools that you utilize inside the pipeline for ETL, model creation, annotation and testing?

Re: Ask HN: What does your ML pipeline look like?

#24

Post by htamas »

tomasdpinho wrote: post #9

As a DevOps Engineer working for a ML-based company and have had worked for others in the past, these are my quick suggestions for production readiness. DOs: If you are doing any kind of soft-realtime (i.e. not batch processing) inference, by exposing a model on a request-response lifecycle, use Tensorflow Serving for concurrency reasons. Version your models and track their training. Use something like MLFlow for tha…

Excuse my ignorance, but why is an NFS better than S3? Both are loading from disk to memory of the Tensorflow Serving container, aren't they?

Re: Ask HN: What does your ML pipeline look like?

#25

Post by maksimum »

tomasdpinho wrote: post #13
hadsed wrote: post #11

Earlier quoted context omitted.

Can you elaborate on why downloading from S3 at startup is a bad idea? And why not synchronous everywhere as opposed to always queues? Good points overall that I'd agree with.

Containers are meant to be stateless infrastructure. By downloading something at startup, you're breaking that contract implicitly. Secondly, depending on where you're deploying, downloads from S3 (and then loading to memory) may take a non-negligible amount of time that can impact the availability of your pods (again, depending on their configuration). Synchronicity everywhere may cause request loss if your ML pipel…

> Containers are meant to be stateless infrastructure. By downloading something at startup, you're breaking that contract implicitly.

I feel that mounting a NFS partition is a similar break of contract. I.e. you could see the same image behave differently depending on what's in the NFS partition. I feel like to get data in a "reproducible" way you need to pull it from a data versioning system. I think there's different ways to implement data versioning with their own trade-offs. NFS and S3, among others, could be used to implement data versioning.

I agree with you that in theory an NFS is more performant because it allows you to load lazily.

Re: Ask HN: What does your ML pipeline look like?

#26

Post by fcolas »

jcims wrote: post #6

Would be interested if anyone in here has a pipeline operating on regulated data (HIPAA, financial, etc). Having a hard time drawing boundaries around what the data science team has access to for development and experimentation vs. where production pipelines would operate. (e.g. where in the process do people/processes get access to live data)

I have a dataset management mechanism (delete, copy, duplicate, etc.) where dataset attributes are tagged as PII (personally identifiable information), and where generic filters are then applied to obfuscate PII for datasets that'll be used by non-privileged users, e.g., data science.

- It's not bullet proof, but it achieves what I'm looking for.

Re: Ask HN: What does your ML pipeline look like?

#27

Post by AndrewKemendo »

mallochio wrote: post #8
AndrewKemendo wrote: post #2

Depends on what you're trying to do. Are you putting a trained inference model into production as a product? Is it a RL system (completely different architecture than an inference system)? Are you trying to build a model with your application data from scratch? Are you doing NLP or CV? As a rule of thumb I look at the event diagram of the application/systems you're trying to implement ML into, which should tell you h…

Thanks for the reply. Could you give some more insight into how and what tools you choose for the different sort of tasks (say NLP vs CV vs RL)? Also, how and why are different tools/pipelines better for production and product building?

How you parse and manage the inputs is significantly different between those types.

With NLP as one example, you need to determine when are you going to do tokenization? - aka break up the inputs into "tokens." So do you do this at ingest, in transit, at rest?

With CV you don't need to do tokenization at all (probably).

So the tools really come out of the use case and how/when you put them into the production chain.

Re: Ask HN: What does your ML pipeline look like?

#28

Post by rpedela »

htamas wrote: post #24
tomasdpinho wrote: post #9

As a DevOps Engineer working for a ML-based company and have had worked for others in the past, these are my quick suggestions for production readiness. DOs: If you are doing any kind of soft-realtime (i.e. not batch processing) inference, by exposing a model on a request-response lifecycle, use Tensorflow Serving for concurrency reasons. Version your models and track their training. Use something like MLFlow for tha…

Excuse my ignorance, but why is an NFS better than S3? Both are loading from disk to memory of the Tensorflow Serving container, aren't they?

NFS is faster and it looks like a normal filesystem to the app so you don't need any special file I/O code.

Re: Ask HN: What does your ML pipeline look like?

#29

Post by Liveanimalcams »

mallochio wrote: post #23
Liveanimalcams wrote: post #19

I'm not a professional but I built a pipeline for Makers Part List - It involves ingesting a video URL, converting the video into images, then storing the images in google storage. Once stored I trigger the model to classify the image. The images are then displayed to annotators who verify/relabel the images. Once I get enough new images the system creates a .csv and uploads it to googles autoML where it retrains my…

Interesting. Could you maybe expand on the tools that you utilize inside the pipeline for ETL, model creation, annotation and testing?

I'm running the front and backend of the consumer site on Heroku. The meat of pipeline is hosted on a DigitalOcean High CPU Droplet. I use ffmpeg to extract images from the provided videos. I store everything in Google Cloud Storage and create references to each photo in Firestore. I use Firebase to power for the image verifying/labeling app I built. Its a simple app that presents the viewer with the image and the label that it was given. If its not correct they enter the correct label. I use a cloud function to move the images into an exportable format for autoML once a new image threshold has been hit. Testing is me using it and seeing if it is correctly identifying the objects.
Post reply on HN