Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My story as a self-taught AI researcher

blog.floydhub.com

Post reply on HN
Print view View original

61–70 of 176 posts

Re: My story as a self-taught AI researcher

#61

Post by ineedasername »

JamesBarney wrote: post #53
ineedasername wrote: post #50

I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understand…

I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?

>in my mind there is also a lot of overlap.

Yep! There can be. But if you want concrete examples, I used Xgboost to identify people within a population at risk for an adverse event. This is strictly #1. If I optimized Xgboost code to make it faster, that's also probably firmly #1. If I improved Xgboost with a better understanding of gradient boosting to provide more accurate results, that's probably a firm case of overlap. When Leo Breiman [0] did his work that led to gradient boosting and tools like Xgboost, that was firmly #2.

[0] https://en.wikipedia.org/wiki/Leo_Breiman

Re: My story as a self-taught AI researcher

#62

Post by K0SM0S »

jph00 wrote: post #34
K0SM0S wrote: post #29

This was a great read (and great nuggets, like that paper on Intelligence by Chollet). I wonder: — Is math a problem for non-academic researchers? Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but ma…

> Is math a problem for non-academic researchers? It takes a while to figure out how to read academic papers, but it's largely about learning the notation. In the end, it maps back to the code you write anyway in most cases, so it's just another way of writing stuff you already know. It's not so much linear algebra you need, since much of that is not relevant to AI. It's really matrix calculus. Which is largely about…

> matrix calculus

It might be wrong but I tend to see vectors and matrices as two notations for the same mathematical object[1]. So I indeed meant matrices! However I didn't see calculus itself as such a big requirement, as it all felt pretty "linear" to me (regressions etc). Are we talking things that e.g. "Calc 2"[2] should cover?

I feel reassured by your first paragraph. This can be done.

I'll definitely work on your tutorial; I assume it's a good benchmark for math pre-requisites in the field. Thanks a lot for the work, and advice.

[1]: That was particularly reinforced with Geometric Algebra, which I'm currently diving in. https://en.wikipedia.org/wiki/Geometric_algebra

[2]: https://www.khanacademy.org/math/calculus-2

Re: My story as a self-taught AI researcher

#63

Post by K0SM0S »

anjc wrote: post #40
jph00 wrote: post #34

Earlier quoted context omitted.

> Is math a problem for non-academic researchers? It takes a while to figure out how to read academic papers, but it's largely about learning the notation. In the end, it maps back to the code you write anyway in most cases, so it's just another way of writing stuff you already know. It's not so much linear algebra you need, since much of that is not relevant to AI. It's really matrix calculus. Which is largely about…

Your tutorial is very good, but to able read even a few paragraphs you need to be proficient with linear algebra and calculus already. > Most papers strike me as requiring a non-trivial knowledge of linear algebra I think this is correct, if you consider college level linear algebra and an intuition for applying it to novel problems to be non-trivial knowledge

> I think this is correct, if you consider college level linear algebra and an intuition for applying it to novel problems to be non-trivial knowledge

Yes, in the context of "a self-taught researcher", I think I intuitively meant anything that precisely requires a degree, typical academic knowledge. E.g. you can become a great business person who won't feel hindered by lack of academic knowledge, you definitely can't do that as a surgeon or lawyer.

I guess I was wondering where math fit in this picture for AI research. (which I should explicitely relate to "#2" in user ineedasername's post, i.e. "AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1 [AI Engineering Research]. What might be termed "basic" or "pure" research.")

Re: My story as a self-taught AI researcher

#64

Post by wigl »

JamesBarney wrote: post #55
wigl wrote: post #39

This reeks of survivorship bias to me. I much prefer Andreas Madsen's more sober and self-conscious take on independent research [0]. > I’d spend 1-2 months completing Fast.ai course V3, and spend another 4-5 months completing personal projects or participating in machine learning competitions... After six months, I’d recommend doing an internship. Then you’ll be ready to take a job in industry or do consulting to se…

> Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020? I don't think the idea is to look for an internship after the course but an additional 4 months of personal projects. After applying state of the art deep learning for 4 months full time you'll have some very cool projects, and you could probab…

I should have been more specific and did not mean to exclude the other 4-5 months in his half year estimate. I meant to say no other background/experience to the position that would qualify the candidate for the role other than fast.ai v3 part 1. I love fast.ai and would still recommend it to all. I just think that the chances for getting into internships in that period, given the difficulties/inefficiencies/biases in the hiring process that even OP has mentioned, are slim.

Re: My story as a self-taught AI researcher

#65

Post by majos »

JamesBarney wrote: post #53
ineedasername wrote: post #50

I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understand…

I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?

One possible distinction is: does the work reveal anything beyond the solution itself? The work might, for example, give one instance of a class of problems for which the tool is useful (bonus points for a formal statement to that effect). Or improve the tool, or improve understanding of the tool’s strengths and weaknesses

I think this is what we try to capture as “expanding human knowledge”.

IMO the more isolated the result (“technique x gave good results for problem y, the end”), the less like “research” it is. Though plenty such papers get into good conferences every year. A nice story and a little reviewer luck go a long way.

Re: My story as a self-taught AI researcher

#66

Post by JamesBarney »

rumanator wrote: post #54
JamesBarney wrote: post #53

Earlier quoted context omitted.

I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?

> Mind providing some concrete examples? It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications. To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those meth…

I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical).

But I was asking because I was specifically looking for concrete examples in deep learning.

Re: My story as a self-taught AI researcher

#67

Post by K0SM0S »

andreilys wrote: post #60
K0SM0S wrote: post #29

This was a great read (and great nuggets, like that paper on Intelligence by Chollet). I wonder: — Is math a problem for non-academic researchers? Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but ma…

“ I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny“ Thanks to transfer learning, tiny datasets are not a major issue to developing AI solutions. Fast.ai makes it super easy to overcome that hurdle.

Major interest spike: nerd-trigger, category "hard problems whose solution exceeds what I have to work with".

Will investigate. Thanks a lot!

Re: My story as a self-taught AI researcher

#68

Post by ineedasername »

AlexCoventry wrote: post #58
ineedasername wrote: post #50

I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understand…

Is AlphaFold 1 or 2?

I'm hardly the official judge of these things, but I would say it depends on how novel of an approach AlphaFold is to the problem. If it's a more efficient tool for doing the same things as before, I would put it towards the #1 end of the spectrum, unless it has also improved our basic understanding of folding or approaches to exploring the solution space of folded proteins, which would shift it towards #2.

Personally I don't know enough about AlphaFold or the problems of protein folding to be remotely confident in my judgment on it

Re: My story as a self-taught AI researcher

#69

Post by JamesBarney »

ineedasername wrote: post #61
JamesBarney wrote: post #53

Earlier quoted context omitted.

I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?

>in my mind there is also a lot of overlap. Yep! There can be. But if you want concrete examples, I used Xgboost to identify people within a population at risk for an adverse event. This is strictly #1. If I optimized Xgboost code to make it faster, that's also probably firmly #1. If I improved Xgboost with a better understanding of gradient boosting to provide more accurate results, that's probably a firm case of ov…

Thanks!

Re: My story as a self-taught AI researcher

#70

Post by jdminhbg »

Gimpei wrote: post #11
Nasrudith wrote: post #7

Earlier quoted context omitted.

It is pretty strange even from an economist really - they of all people should be able to understand and articulate the difference between signaling value and direct utility value of a given good or service.

It's not a majority view among economists. Caplan is the only person I can think of who holds this view.

Caplan is definitely not the only economist who holds this view. Most place the signaling/human capital split around 50/50: https://www.econlib.org/archives/2011/11/kauffman_econ_b.htm...
Post reply on HN