Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End-to-end implementation of a machine learning pipeline (2017)

spandan-madan.github.io

Post reply on HN
Print view View original

41–45 of 45 posts

Re: End-to-end implementation of a machine learning pipeline (2017)

#41

Post by crucialfelix »

tamersalama wrote: post #31
roystonvassey wrote: post #16

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

Could that be because of using Jupyter notebook itself? I like Jupyter for data and machine learning 'journalism', but I don't see it as the a proper medium to address the 'last-mile'. The insights driven from Jupyter, in my opinion, are not actionable and well integrated enough. It is becoming a de-facto medium reminding me of shared Excel files.

Yes, Jupyter is for initial exploration. Then you write solid normal production code. Then you might write further notebooks that import that production code and run/visualize metrics and reporting for your client (probably non-technical people).

I had a "data scientist" submit notebooks to us as if we could ship any of that in production. (We fired him.) It's for hacking and blogging, not for production work.

Re: End-to-end implementation of a machine learning pipeline (2017)

#42

Post by bosie »

Radim wrote: post #36
tamersalama wrote: post #31

Earlier quoted context omitted.

Could that be because of using Jupyter notebook itself? I like Jupyter for data and machine learning 'journalism', but I don't see it as the a proper medium to address the 'last-mile'. The insights driven from Jupyter, in my opinion, are not actionable and well integrated enough. It is becoming a de-facto medium reminding me of shared Excel files.

Could be. Using Jupyter for ML development or even prototyping (as opposed to presentations / demonstration / teaching like the OP — that's where Jupyter really shines) is a red flag. I see a similar pattern with Pandas: some people use Pandas not because it's the right tool for the job (Pandas has many strengths), but because they're scared of writing comprehension loops and basic data structures. To avoid the CS-y…

How do you suggest prototyping without Jupyter? (in case prototyping means researching an approach)

Re: End-to-end implementation of a machine learning pipeline (2017)

#45

Post by Reebz »

heinrichhartman wrote: post #35

Did they have to put a big "Harvard University" banner at the top of the GitHub repo: https://github.com/Spandan-Madan/DeepLearningProject ? This is a private repository, right? Is the code owned by Harvard? For me personally, I find this off-putting. Let your content speak for itself. No added credibility when the affiliation is advertised like this. Content looks good, though! :)

Yes, this confused me too. Especially considering Spandan Madan is an MIT researcher, I don't understand the overt branding of a different school
Post reply on HN