Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End-to-end implementation of a machine learning pipeline (2017)

spandan-madan.github.io

Post reply on HN
Print view View original

31–40 of 45 posts

Re: End-to-end implementation of a machine learning pipeline (2017)

#31

Post by tamersalama »

roystonvassey wrote: post #16

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

Could that be because of using Jupyter notebook itself? I like Jupyter for data and machine learning 'journalism', but I don't see it as the a proper medium to address the 'last-mile'. The insights driven from Jupyter, in my opinion, are not actionable and well integrated enough. It is becoming a de-facto medium reminding me of shared Excel files.

Re: End-to-end implementation of a machine learning pipeline (2017)

#32

Post by tonyarkles »

mattkrause wrote: post #26
aportnoy wrote: post #10

Earlier quoted context omitted.

A more Pythonic way do to this would be id_to_name = {g['id']: g['name'] for g in list_of_genres} And for i in range(len(list_of_genres)) is really a dangerous antipattern better replaced with for genre in list_of_genres:

Out of curiosity, how is that dangerous ?

My guess is based on Python 2, where range(n) would return a fully inflated list of 0..(n-1). If len(x) is large, you could be allocating giant temporary lists just to iterate through them once. Using xrange() was the Python 2 solution (it would return a generator instead), but, if I recall correctly, Python 3 fixed this s.t. range() returns a generator.

Re: End-to-end implementation of a machine learning pipeline (2017)

#33

Post by adamb »

sidlls wrote: post #30
roystonvassey wrote: post #16

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

Seconding this. I have run a data science and machine learning team for the last couple of years. By far the most challenging part of our work has been convincing our data management team that we aren't just another front end widget factory and our development/operations staff that we aren't choosing "non-standard" tech to deliver model results into production. The model maintenance is difficult, too, due to poor dat…

What have you found to work best when coordinating with your data management and development/operations staff?

Re: End-to-end implementation of a machine learning pipeline (2017)

#34

Post by rasmi »

DrNuke wrote: post #23
roystonvassey wrote: post #16

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

That’s it, really. Any good reference to keep up to date with the last-mile best practices for the average ML practitioner? Thanks!

I link these resources often, but they are often relevant! See "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" [1] and the Rules of Machine Learning [2]. Another classic: "Machine Learning: The High Interest Credit Card of Technical Debt" [3], and recently added: Responsible AI Practices [4].

[1] https://ai.google/research/pubs/pub46555

[2] https://developers.google.com/machine-learning/rules-of-ml/

[3] https://ai.google/research/pubs/pub43146

[4] https://ai.google/education/responsible-ai-practices

Re: End-to-end implementation of a machine learning pipeline (2017)

#35

Post by heinrichhartman »

Did they have to put a big "Harvard University" banner at the top of the GitHub repo: https://github.com/Spandan-Madan/DeepLearningProject ? This is a private repository, right? Is the code owned by Harvard?

For me personally, I find this off-putting. Let your content speak for itself. No added credibility when the affiliation is advertised like this.

Content looks good, though! :)

Re: End-to-end implementation of a machine learning pipeline (2017)

#36

Post by Radim »

tamersalama wrote: post #31
roystonvassey wrote: post #16

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

Could that be because of using Jupyter notebook itself? I like Jupyter for data and machine learning 'journalism', but I don't see it as the a proper medium to address the 'last-mile'. The insights driven from Jupyter, in my opinion, are not actionable and well integrated enough. It is becoming a de-facto medium reminding me of shared Excel files.

Could be. Using Jupyter for ML development or even prototyping (as opposed to presentations / demonstration / teaching like the OP — that's where Jupyter really shines) is a red flag.

I see a similar pattern with Pandas: some people use Pandas not because it's the right tool for the job (Pandas has many strengths), but because they're scared of writing comprehension loops and basic data structures. To avoid the CS-y stuff. But without the CS-y stuff, the result ends up a mess of lambdas, weird reindexing and buggy copy/view semantics.

And then "the next guy", the one who's job it is to clean up and productionalize the maverick's output, ends up having to reinvent and fix the entire solution. Basically doing both jobs.

Re: End-to-end implementation of a machine learning pipeline (2017)

#37

Post by sandGorgon »

adamb wrote: post #33
sidlls wrote: post #30

Earlier quoted context omitted.

Seconding this. I have run a data science and machine learning team for the last couple of years. By far the most challenging part of our work has been convincing our data management team that we aren't just another front end widget factory and our development/operations staff that we aren't choosing "non-standard" tech to deliver model results into production. The model maintenance is difficult, too, due to poor dat…

What have you found to work best when coordinating with your data management and development/operations staff?

same question here. im keen to understand this. Especially around responsibilities, OKRs and KRAs

Re: End-to-end implementation of a machine learning pipeline (2017)

#38

Post by fsavard »

DrNuke wrote: post #23
roystonvassey wrote: post #16

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

That’s it, really. Any good reference to keep up to date with the last-mile best practices for the average ML practitioner? Thanks!

It's a theme I like to read about (I mean "practical issues around ML in production settings"). I find company blogs and some research publications are great resources. Examples:

- https://eng.uber.com/ - https://code.fb.com/

and many more. Google also publishes papers on various engineering practices obviously, some ML-related, but I can't find a blog where they focus on that specifically.

Also it's not "to keep up to date", but there's a great paper (from Google) that's often cited:

Machine Learning: The High Interest Credit Card of Technical Debt https://ai.google/research/pubs/pub43146

It talks about issues you face over the long run (I've experienced some of those). It also provides interesting pointers for further reading, e.g. about "pipeline jungles".

If others have pointers, I'm curious to hear about them as well.

Re: End-to-end implementation of a machine learning pipeline (2017)

#39

Post by Octokat »

jamesblonde wrote: post #14

This is a very good way to get started building ML pipelines. When you do it at scale, you often need to use a broader range of tools. Here's how we do it in Hopsworks with Python the whole way (using Airflow to orchestrate the different steps): https://hops.readthedocs.io/en/latest/hopsml/hopsML.html

Your pipeline design looks pretty sweet!

Re: End-to-end implementation of a machine learning pipeline (2017)

#40

Post by sidlls »

adamb wrote: post #33
sidlls wrote: post #30

Earlier quoted context omitted.

Seconding this. I have run a data science and machine learning team for the last couple of years. By far the most challenging part of our work has been convincing our data management team that we aren't just another front end widget factory and our development/operations staff that we aren't choosing "non-standard" tech to deliver model results into production. The model maintenance is difficult, too, due to poor dat…

What have you found to work best when coordinating with your data management and development/operations staff?

Every organization and team is different. Often I've found two approaches work best: going around the road blocks and managing everything end to end, then getting buy-in for data and ops to own it properly after the fact (playing up the political angle of owning more stuff after we do the heavy lifting), and the brute force method of just meeting after meeting to educate people about the differences in use cases and deployment for ML products.
Post reply on HN