Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Little Learner: A Straight Line to Deep Learning

mitpress.mit.edu

Post reply on HN
Print view View original

71–80 of 102 posts

Re: The Little Learner: A Straight Line to Deep Learning

#71

Post by rg111 »

rosetremiere wrote: post #60
rg111 wrote: post #57

Earlier quoted context omitted.

I think all serious researchers have implemented core Deep Learning algorithms from scratch. I know I have. There are two books that do exactly this: 1. Deep Learning from Scratch 2. Data Science from Scratch In these books, you implement each part of the ML/DL pipeline, from scratch, in Python. There is also a GitHub project call Minitorch that teaches you the inner workings of a framework like PyTorch. And then the…

How tied are those two books to python? If very much, are there books covering the same content with a different programming language?

They are aren’t that tied to Python.

Even if you think so, Python is really an easy language, and you can easily port the code to something else.

If you already have the basic ideas about the parts of a Neural Network pipeline, you can just search google "implement part-X in Y language", and you will get well written articles/tutorials.

Many learners/practitioners of Deep Learning, when they have the big enough picture, write an NN training loop in their favorite language(s) and post it online. I remember seeing a good enough "Neural Network in APL" playlist in YT. It implements every piece in APL and gains like 90%+ accuracy in MNIST.

I also remember seeing articles in Lisp (of course!), C, Elixir, and Clojure.

I am writing one in J-lang in my free time.

Re: The Little Learner: A Straight Line to Deep Learning

#72

Post by hedgehog0 »

havercosine wrote: post #66
mattfrommars wrote: post #45

Someone who has access to O'Reilly and other free source to learn ML/AI e.g. Fast.ai, is there any advantage of buying this?

'Little..' series books are famous for their pedagogical style: small bit of concepts, presented as a dialogue, making you think at each step, in a well thought out order. It will appeal to anyone who wants to build it out themselves, understand the 'big' ideas in the field. Skim through the second chapter to see how this will be done: they define line function in inverted way where slope and intercept are taken as p…

So you mean that fast.ai maybe a better resource for people new to general ML?

Re: The Little Learner: A Straight Line to Deep Learning

#74

Post by albertzeyer »

rg111 wrote: post #69
albertzeyer wrote: post #65

Earlier quoted context omitted.

What does "from scratch" really mean? You don't reimplement Python itself, or invent a new GPU hardware, a new CUDA including compiler, etc. You don't reimplement the OS. Where do you draw the line? Do you reimplement matmul or other basics? Do you reimplement auto-diff? Maybe PyTorch or TensorFlow using auto-diff is a good "from scratch" basepoint, without using predefined optimizers, or modules/layers, or anything.…

I don't understand why you are arguing with me. Yes, in those books, you do implement matmul, auto-diff, etc.

I don't understand. I don't argue with you? I also don't speak about these books. I just made a generic comment, to start a discussion.

I just wanted to point out that "from scratch" is not really well defined. There is always some arbitrary line. I just found it interesting to discuss and think about where to draw this line exactly. Obviously it's never really from scratch, i.e. you don't reinvent the hardware level, for example. Or you don't start with teaching quantum physics. So you start from somewhere.

And I was wondering whether auto-diff is maybe something which could also already be the starting point, or also matmul. Reimplementing an efficient matmul on CUDA is not easy, and might distract from the main deep learning content. But it depends also where you want to have the focus on.

Re: The Little Learner: A Straight Line to Deep Learning

#75

Post by agumonkey »

bigdict wrote: post #42
zelphirkalt wrote: post #41

Earlier quoted context omitted.

Some time ago I wrote this: https://notabug.org/ZelphirKaltstahl/guile-ml -- It contains a parallelized decision tree implementation in GNU Guile, a Scheme dialect. There is also https://github.com/lloda/guile-ffi-cblas which at least gives fast matrix multiplication. Example usage I have here: https://notabug.org/ZelphirKaltstahl/guile-math-examples/src... I don't know what is in the book, but things are possible.

For sure, Lecun wrote a DL framework in Lisp decades ago.

I didn't know about his past feat. Couldn't find the lisp framework you mention but there's this https://lush.sourceforge.net/faq.html

(trivia, he's also behind djvu format)

Re: The Little Learner: A Straight Line to Deep Learning

#76

Post by rg111 »

albertzeyer wrote: post #74
rg111 wrote: post #69

Earlier quoted context omitted.

I don't understand why you are arguing with me. Yes, in those books, you do implement matmul, auto-diff, etc.

I don't understand. I don't argue with you? I also don't speak about these books. I just made a generic comment, to start a discussion. I just wanted to point out that "from scratch" is not really well defined. There is always some arbitrary line. I just found it interesting to discuss and think about where to draw this line exactly. Obviously it's never really from scratch, i.e. you don't reinvent the hardware level…

Now it makes more sense.

Maybe I misread your comment.

Thanks for rewording.

Where do people draw the line? Where they want to.

Some people choose to go closer to the metal than the others. It's just people's choice.

Some just write stuff in Python, some write CUDA kernel for themselves (some among them had to), a friend even wrote his own compiler and programming language for Deep Learning.

So it depends on your choice. And how much deep you want to go also depends on what you want to do- i.e. you choice of career, direction of research, etc.

Re: The Little Learner: A Straight Line to Deep Learning

#77

Post by greymalik »

bzhang255 wrote: post #31
sn9 wrote: post #28

Earlier quoted context omitted.

The great thing about Scheme is that you can learn it in less than 10 minutes and focus on the ideas.

Sure - but I think that's the least important part of my comment. The preview is basically just an intro to Scheme. I would rather be able to see how well the book covers interesting ideas about deep learning. While the jury is still out, my personal skepticism is that this is a really beautiful passion project that looks great on bookshelves, but that any serious learner would be better served dedicating their time…

Can you suggest any?

Re: The Little Learner: A Straight Line to Deep Learning

#78

Post by joxel »

greymalik wrote: post #77
bzhang255 wrote: post #31

Earlier quoted context omitted.

Sure - but I think that's the least important part of my comment. The preview is basically just an intro to Scheme. I would rather be able to see how well the book covers interesting ideas about deep learning. While the jury is still out, my personal skepticism is that this is a really beautiful passion project that looks great on bookshelves, but that any serious learner would be better served dedicating their time…

Can you suggest any?

[dead]

Re: The Little Learner: A Straight Line to Deep Learning

#79

Post by freilanzer »

freilanzer wrote: post #68
fithisux wrote: post #63

Earlier quoted context omitted.

Will do. I have also found that 3 days ago https://github.com/zdimension/hm-infer-scheme fantastic for research

CL is decent in this regard: http://alhassy.com/TypedLisp

I meant regarding types, I think the equivalent to the other link would be Coalton: https://github.com/coalton-lang/coalton
Post reply on HN