Live data from Hacker News

End-to-end implementation of a machine learning pipeline (2017)

spandan-madan.github.io

21–30 of 45 posts

Re: End-to-end implementation of a machine learning pipeline (2017)

#21
post #3

An earlier discussion: https://news.ycombinator.com/item?id=14781888

Yup! That version was in Keras. It's now been re-written in PyTorch as well! Thanks to https://github.com/AnshulBasia .

Any update on the NLP tutorial? I keep checking this repo [0] but seems it hasn't been updated lately. I hope you didn't abandon this project

[0] - https://github.com/Spandan-Madan/NLP-Intuition-and-Applicati...

Re: End-to-end implementation of a machine learning pipeline (2017)

#22

Earlier quoted context omitted.

Yup! That version was in Keras. It's now been re-written in PyTorch as well! Thanks to https://github.com/AnshulBasia .

Any update on the NLP tutorial? I keep checking this repo [0] but seems it hasn't been updated lately. I hope you didn't abandon this project [0] - https://github.com/Spandan-Madan/NLP-Intuition-and-Applicati...

https://news.ycombinator.com/item?id=18298670

Re: End-to-end implementation of a machine learning pipeline (2017)

#23

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

That’s it, really. Any good reference to keep up to date with the last-mile best practices for the average ML practitioner? Thanks!

Re: End-to-end implementation of a machine learning pipeline (2017)

#24
It's interesting, however the webpage seems broken on Firefox Android (latest, Android 8.1) :

- some values in the command outputs dont match the author's comments (or maybe I misunderstood some?)

- there are some big red blocks of errors in the outputs

- the outputs of the trainings are way too verbose for mobile reading

I guess they are issues on Jupyter's framework side. It would be nice if mobile were treated as first-class viewer.

Re: End-to-end implementation of a machine learning pipeline (2017)

#25
post #9

Earlier quoted context omitted.

Precisely. I strongly believe that the purpose of tutorials is to be inclusive of all people. That's something I realized as a TA, making things explicit never hurts. There's always someone who can gain from more detail :)

I'm going to throw out a plug for that mindset going beyond tutorials. It is a suspect proposition that anything is gained by turning 3 lines of code into one line of code. Unless it is javascript for the Google homepage or somesuch where the bytes matter. Moving code from a bad data model to a good one usually correlates with a big reduction in line count, but the gain is in choosing more appropriate data structures…

Readability of the code is always relative to the reader. In any language that I've mastered, I've always preferred more verbose options at first, but with experience I found out that I find condensed versions to be more elegant and time-saving.

In the end, choose code style that matches level of your audience. If you write a one-off python script that automizes some build process in mostly non-python codebase, it should probably be very verbose and easy to understand. If, on the other hand, you're writing code in a decently advanced codebase and most of your colleagues are fluent in the language (or at least, supposed to be), it makes sense to use as much condensed syntax sugar as possible.

Re: End-to-end implementation of a machine learning pipeline (2017)

#26

Earlier quoted context omitted.

Is your code intentionally verbose (for the sake of being explicit)? It seems like it could be condensed a lot by using Pythonic structures. For example you could replace block 39 with a one-liner: Genre_ID_to_name=dict([(g['id'], g['name']) for g in list_of_genres]) In other places, you would benefit a lot from the enumerate(..) function, which returns (index, item) tuples when called on a list.

A more Pythonic way do to this would be id_to_name = {g['id']: g['name'] for g in list_of_genres} And for i in range(len(list_of_genres)) is really a dangerous antipattern better replaced with for genre in list_of_genres:

Out of curiosity, how is that dangerous?

Re: End-to-end implementation of a machine learning pipeline (2017)

#27

Earlier quoted context omitted.

Is your code intentionally verbose (for the sake of being explicit)? It seems like it could be condensed a lot by using Pythonic structures. For example you could replace block 39 with a one-liner: Genre_ID_to_name=dict([(g['id'], g['name']) for g in list_of_genres]) In other places, you would benefit a lot from the enumerate(..) function, which returns (index, item) tuples when called on a list.

Precisely. I strongly believe that the purpose of tutorials is to be inclusive of all people. That's something I realized as a TA, making things explicit never hurts. There's always someone who can gain from more detail :)

I agree with the sentiment in general, but that example from the OP is actually pretty clear, no? Some might argue it is at least as clear as the code it is intended to replace.

Re: End-to-end implementation of a machine learning pipeline (2017)

#28

Earlier quoted context omitted.

Is your code intentionally verbose (for the sake of being explicit)? It seems like it could be condensed a lot by using Pythonic structures. For example you could replace block 39 with a one-liner: Genre_ID_to_name=dict([(g['id'], g['name']) for g in list_of_genres]) In other places, you would benefit a lot from the enumerate(..) function, which returns (index, item) tuples when called on a list.

A more Pythonic way do to this would be id_to_name = {g['id']: g['name'] for g in list_of_genres} And for i in range(len(list_of_genres)) is really a dangerous antipattern better replaced with for genre in list_of_genres:

[deleted]

Re: End-to-end implementation of a machine learning pipeline (2017)

#29

Earlier quoted context omitted.

Is your code intentionally verbose (for the sake of being explicit)? It seems like it could be condensed a lot by using Pythonic structures. For example you could replace block 39 with a one-liner: Genre_ID_to_name=dict([(g['id'], g['name']) for g in list_of_genres]) In other places, you would benefit a lot from the enumerate(..) function, which returns (index, item) tuples when called on a list.

Precisely. I strongly believe that the purpose of tutorials is to be inclusive of all people. That's something I realized as a TA, making things explicit never hurts. There's always someone who can gain from more detail :)

Couldn't agree more. Coming from a world where I strive to come with great self explanatory naming conventions, Python code most often looks minified to me, and I have a hard time reading it...

Re: End-to-end implementation of a machine learning pipeline (2017)

#30

Great explanation and I love the fact that the entire presentation is a Jupyter Notebook! A non-academic observation - the 'real-world' challenge of ML pipelines is what I call the 'last-mile' problem of ML - operationalizing your model. You begin to run into problems of: 1. How often do you 'score' live data? How will this affect latency, data ingestion etc? 2. How often do you have to update your weights, if you wa…

Seconding this. I have run a data science and machine learning team for the last couple of years. By far the most challenging part of our work has been convincing our data management team that we aren't just another front end widget factory and our development/operations staff that we aren't choosing "non-standard" tech to deliver model results into production. The model maintenance is difficult, too, due to poor data management practices but it's less challenging than the other items for my team.
Post reply on HN