Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Deep Learning

udlbook.github.io

Post reply on HN
Print view View original

91–100 of 103 posts

Re: Understanding Deep Learning

#91

Post by komatsu »

contrarian1234 wrote: post #59

It's very hard to judge a book like this... (based on a table of contents?) Who is the author ? Have they published anything else highly rated ? Are there good reviews from people that know what they're talking about? Are there good reviews from students that don't know anything ?

I can highly recommend the author. His last book "Computer Vision: Models, Learning, and Inference" is very readable, approaches the matter from unorthodox viewpoints + includes lot of excellent figures supporting the text. I'm buying this on paper!

Re: Understanding Deep Learning

#92

Post by mindcrime »

trace_id wrote: post #38
ldjkfkdsjnv wrote: post #8

Earlier quoted context omitted.

Better to understand the bounds of whats currently possible. And then recognize when that changes. Much more economically valuable

Even better: change the bounds of whats possible ;)

> Even better: change the bounds of whats possible ;)

... which will be easier it you have a solid grasp of the foundations of the field. If you only ever focus on the "latest shiny" you'll be lost and left floundering when the landscape changes out from underneath you.

Re: Understanding Deep Learning

#93

Post by mindcrime »

PeterisP wrote: post #18
water-your-self wrote: post #5

No chapter on RNNs, but one on transformers is interesting, having last read Deep learning by ian goodfellow in 2016

RNNs have "lost the hardware lottery" by being structurally not that efficient to train on the cost-effective hardware that's available. So they're not really used for much right now - though IMHO they are conceptually sufficiently interesting enough to cover in such a course.

> RNNs have "lost the hardware lottery" by being structurally not that efficient to train on the cost-effective hardware that's available.

Which suggests two obvious paths forward:

1. Don't bother learning / using RNN's

2. Co-develop new hardware / new RNN architectures that work together to provide great performance per unit of price.

Now of course nobody is saying (well, I am not saying) that (2) would be easy... or even necessarily possible. But somebody should at least be intrigued by the idea. And in the world we live in today where FPGA's and other devices make it easier than ever to experiment with custom hardware architectures... it might be worth taking a stab at it.

Re: Understanding Deep Learning

#94

Post by nextos »

fastneutron wrote: post #84
nextos wrote: post #34

Earlier quoted context omitted.

This presentation from Deep Mind outlines some foundational ML books: https://drive.google.com/file/d/1lPePNMGMEKoaDvxiftc8hcy-rFp... For the impatient, look into slide #123. Essentially, the recommendations are Murphy, Gelman, Barber, and Deisenroth. Note these slides have a Bayesian bias. In spite of that, Murphy is a great DL book. Besides, going through GLMs is a great way to get into DL.

Reality has a well known Bayesian bias… Joking aside, these slides are excellent! Is there an associated video or course that they were a part of?

No, these were part of a conference held at Tubingen in 2020.

Re: Understanding Deep Learning

#95

Post by calf »

adamnemecek wrote: post #17

All machine learning is Hopf convolution, analogous to renormalization. This should come as no surprise, renormalization can be modeled via the Ising model which itself is closely related to Hopfield networks which are recurrent networks.

That's an interesting point, are there any resources to learn about this? I have a CS background, in that we generally only cover 1st-year physics and very little theoretical math beyond linear algebra, etc.

Re: Understanding Deep Learning

#96

Post by braza »

martingoodson wrote: post #68

Most comments here are in one of two camps: 1) you don't need to know any of this stuff, you can make AI systems without this knowledge, or 2) you need this foundational knowledge to really understand what's going on. Both perspectives are correct. The field is bifurcating into two different skill sets: ML engineer and ML scientist (or researcher). It's great to have both types on a team. The scientists will be too s…

> But when they hit a roadblock or need to adapt an algorithm many engineers will stumble.

My experience is the other way around.

People underestimate how powerful building systems is and how most of the problems worth solving are boring and require out-of-the-box techniques.

During the last decade, I was in some teams and I noticed the same pattern: The company has some extra budget and "believes" that their problem is exceptional.

Then goes and hires some PhDs Data scientists with some publications but only know R and are fresh from some Python bootcamps.

After 3 months, or this new team no much was done, tons of Jupyter notebooks around but no code in production, and some of them did not even have an environment to do experimentation.

The business problem is still not solved. The company realizes that having a lot of Data Scientists not not so many Data/ML Enginers means that they are (a) blocked to do pushing something to production or (b) are creating a death star of data pipelines + algorithms + infra (spending 70% more of resources due to lack of knowledge on Python).

The project gets delayed. Some people become impatient.

Now you have a solid USD 2.5 million/year team that is not capable of delivering a proof of concept due to the fact that people cannot do the serving via Batch or via REST API.

The company lost momentum, competitors moved fast. They released an imperfect solution, but a solution ahead, and they have users on it and they are enhancing.

Frustration kicks in, and PMs and Eng Managers fight about accountability. VP of Product and Engineering wants heads in a silver plate.

Some PhDs get fired and go to be teachers in some local university.

Fin.

Re: Understanding Deep Learning

#97

Post by adamnemecek »

calf wrote: post #95
adamnemecek wrote: post #17

All machine learning is Hopf convolution, analogous to renormalization. This should come as no surprise, renormalization can be modeled via the Ising model which itself is closely related to Hopfield networks which are recurrent networks.

That's an interesting point, are there any resources to learn about this? I have a CS background, in that we generally only cover 1st-year physics and very little theoretical math beyond linear algebra, etc.

There is some yeah, there is a recent book on deep learning via renormalization https://arxiv.org/pdf/2106.10165.pdf.

I have a discord https://discord.cofunctional.ai

Re: Understanding Deep Learning

#98

Post by mvkel »

GeneralMayhem wrote: post #41
nsxwolf wrote: post #9

As someone who missed the boat on this, is learning about this just for historical purposes now, or is there still relevance to future employment? I just imagine the OpenAI eats everyone's lunch in regards to anything AI related, am I way off base?

The most important thing to learn for most practical purposes is what the thing can actually do. There's a lot of fuzzy thinking around ML - "throw AI at it and it'll magically get better!" Sources like Karpathy's recent video on what LLMs actually do are good anti-hype for the lay audience, but getting good practical working knowledge that's a level deeper is tough without working through it. You don't have to memor…

Awesome response, and a reasoned take.

Re: Understanding Deep Learning

#99

Post by siddbudd »

contrarian1234 wrote: post #59

It's very hard to judge a book like this... (based on a table of contents?) Who is the author ? Have they published anything else highly rated ? Are there good reviews from people that know what they're talking about? Are there good reviews from students that don't know anything ?

Some Google-fu for you:

based on a table of contents? You can download the draft of Chapters 1-21 (500+ pages) from the linked site.

Who is the author ? Simon J. D. Prince is Honorary Professor of Computer Science at the University of Bath and author of Computer Vision: Models, Learning and Inference. A research scientist specializing in artificial intelligence and deep learning, he has led teams of research scientists in academia and industry at Anthropics Technologies Ltd, Borealis AI, and elsewhere.

Have they published anything else highly rated ? Author of >50 peer reviewed publications in top tier conferences (CVPR, ICCV, SIGGRAPH etc.) https://scholar.google.com/citations?user=fjm67xYAAAAJ&hl=en

Are there good reviews [...] The book has not been published, this is literally a free draft that you are looking at. The book is listed on Amazon as a pre-order for 85USD.

Re: Understanding Deep Learning

#100

Post by johnthescott »

martingoodson wrote: post #85
mi_lk wrote: post #79

Earlier quoted context omitted.

I guess this message is delivered by an AI scientist, sure. It's almost self-exploratory that when you hit a roadblock in practice you go back to foundations, and good people should aim to do both. In that case I don't see where ML engineer/scientist bifurcation comes from except for some to feel good about themselves

Not at all. It's something I've seen in practice over many years. Neither skill set is 'better' than the other, just different. There is a need for people who are able to build using available tools, but who don't have an interest in the theory or foundations of the field. It's a valuable mindset and nothing in my original comment suggested otherwise. It's also pretty clear that many comments on this post divide into…

as a friend from at&t dallas told me, tis cheaper to turn a mathematician into a programmer than a programmer into a mathematician.
Post reply on HN