End-to-end implementation of a machine learning pipeline (2017) • Hacker News Board

Re: End-to-end implementation of a machine learning pipeline (2017)

#11

Post by abhishekjha » Thu, Oct 25, 2018, 6:55 AM UTC

mario0b1 wrote: post #8

spandan-madan wrote: post #2

For feature requests on this, please create an issue on the github Repo! For future tutorial suggestions, mail me at smadan@mit.edu. A new one on NLP is coming soon!

Just started an ML course this semester. I am not sure if I even have time to use this as additional resource, but it looks super awesome after skimming through it. Definitly going into my favorites and if I don't use it as additional resource now, I will read it later. Thanks for making all this work public!

Does the ML course have videos that can be accesed online?

Re: End-to-end implementation of a machine learning pipeline (2017)

#12

Post by barrystaes » Thu, Oct 25, 2018, 6:58 AM UTC

roenxi wrote: post #9

spandan-madan wrote: post #5

Earlier quoted context omitted.

Precisely. I strongly believe that the purpose of tutorials is to be inclusive of all people. That's something I realized as a TA, making things explicit never hurts. There's always someone who can gain from more detail :)

I'm going to throw out a plug for that mindset going beyond tutorials. It is a suspect proposition that anything is gained by turning 3 lines of code into one line of code. Unless it is javascript for the Google homepage or somesuch where the bytes matter. Moving code from a bad data model to a good one usually correlates with a big reduction in line count, but the gain is in choosing more appropriate data structures…

I agree, gains should go to reading the code most of the time. But when you say

> It is a suspect proposition that anything is gained by turning 3 lines of code into one line of code.

Some code is way more verbose than its description would be. A named function signals intent:

  function multidimensional_transform(the_data)

And for Javascript ES2015 there is new syntax (like spread operators) that improves code the same way.

Re: End-to-end implementation of a machine learning pipeline (2017)

#13

Post by lolitan » Thu, Oct 25, 2018, 6:59 AM UTC

thank you guys

Re: End-to-end implementation of a machine learning pipeline (2017)

#14

Post by jamesblonde » Thu, Oct 25, 2018, 7:02 AM UTC

This is a very good way to get started building ML pipelines. When you do it at scale, you often need to use a broader range of tools. Here's how we do it in Hopsworks with Python the whole way (using Airflow to orchestrate the different steps):

https://hops.readthedocs.io/en/latest/hopsml/hopsML.html

Re: End-to-end implementation of a machine learning pipeline (2017)

#15

Post by zdk » Thu, Oct 25, 2018, 7:17 AM UTC

As a beginner and a self-tought. This style of tutorial is practically good and persuasive for practitioners. Thank you very much for investing time to do this.

Re: End-to-end implementation of a machine learning pipeline (2017)

#16

Post by roystonvassey » Thu, Oct 25, 2018, 7:32 AM UTC

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook!

A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of:

1. How often do you 'score' live data? How will this affect latency, data ingestion etc?

2. How often do you have to update your weights, if you want your model's performance to be consistent?

3. Integration with source systems

4. If you build your final scoring model on library-dependent languages like Python, how do you ensure no breakages? (Docker solves this to a large extent though)

Re: End-to-end implementation of a machine learning pipeline (2017)

#17

Post by stuartaxelowen » Thu, Oct 25, 2018, 8:05 AM UTC

aportnoy wrote: post #10

d_burfoot wrote: post #4

Earlier quoted context omitted.

Is your code intentionally verbose (for the sake of being explicit)? It seems like it could be condensed a lot by using Pythonic structures. For example you could replace block 39 with a one-liner: Genre_ID_to_name=dict([(g['id'], g['name']) for g in list_of_genres]) In other places, you would benefit a lot from the enumerate(..) function, which returns (index, item) tuples when called on a list.

A more Pythonic way do to this would be id_to_name = {g['id']: g['name'] for g in list_of_genres} And for i in range(len(list_of_genres)) is really a dangerous antipattern better replaced with for genre in list_of_genres:

And if you need the index:

    for idx, genre in enumerate (list_of_genres)

Re: End-to-end implementation of a machine learning pipeline (2017)

#18

Post by ratsimihah » Thu, Oct 25, 2018, 8:50 AM UTC

This is great, most tutorials out there assume the dataset already exists. Nice move covering the entire pipeline!

Also, looks like that notebook needs an update.

https://github.com/ContinuumIO/anaconda-issues/issues/6678

Re: End-to-end implementation of a machine learning pipeline (2017)

#19

Post by junke » Thu, Oct 25, 2018, 10:00 AM UTC

Note: at the beginning, you speak about learning a function "g" that approximates a function "f", then later, you swap them and learn a function "f" that approximates "g". That could be confusing;

Re: End-to-end implementation of a machine learning pipeline (2017)

#20

Post by [deleted] » Thu, Oct 25, 2018, 10:27 AM UTC

spandan-madan wrote: post #6

spandan-madan wrote: post #5

Earlier quoted context omitted.

Precisely. I strongly believe that the purpose of tutorials is to be inclusive of all people. That's something I realized as a TA, making things explicit never hurts. There's always someone who can gain from more detail :)

Also, the code for the PyTorch version has been contributed by https://github.com/AnshulBasia . But it is basically a port of my original version in Keras, which was equally verbose :)

[deleted]