Was anyone else really put off by the congratulatory tone of the article, and the #Quirks list on the resume?:
https://github.com/emilwallner/Emil-Wallner-LinkedIn-Resume#...
My story as a self-taught AI researcher
51–60 of 176 posts
Re: My story as a self-taught AI researcher
#52Earlier quoted context omitted.
It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us. The usual corollary (that ML should "…
"The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself." Yes, and they do. They aren't choosing completely arbitrary algorithms when they attempt to solve a ML problem, they are typically using approaches that have already be…
Indeed, but what's the magnitude?
The question is, how much information is encoded in those algos (to me, low-order logical truths about a few elementary variables, low degree of freedom for the system overall), compared to how much information is encoded in the "algos of the human brain" (and actually the whole body, if we admit that intelligence has little motivation to emerge if there's no signal to process and no action to ever be taken).
I was merely pointing out this outstanding asymmetry, as I see it, and the unfairness of judging our AI progress (or setting goals for it) relatively to anything even remotely close to evolved species, in terms of end-result behavior, emergent high-level observations.
Think of it this way: a tiny neural net (equivalent to the brain of what, not even an insect?) "generationally evolved" enough by us to be able to recognize cats and license numbers and process human speech and suggest songs and whatnot is really not too shabby. I'd call it monumental successs to be able to focus a NN so well on a vertical skill. But that's also low-order low-freedom, in the grander scheme of things, and "focus" (verticality) is just one aspect of intelligence (e.g. the raging battle is for "context" as we speak, horizontality and sequentiality of knowledge; and you can see how the concept of "awareness", even just mechanical, lies behind that). So, many more steps to go. So vastly much more to encode in our models before they're able to take a lesson in one standing and a few examples.
It really took big-big-big data for evolution to do it, anyway, and we're speeding that up thanks to focus in design, and electronics to hasten information processing, but not fundamentally changing the law of neural evolution, it seems.
If you ask me, the next step is to encode structural information in the neuron itself, as a machine or even network thereof, because that's how biology does it (the "dumb" logic gate transistor model is definitely wrong on all accounts, too simplistic). Seems like the next obvious move, architecturally.
Re: My story as a self-taught AI researcher
#53I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understand…
I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1.
But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?
Re: My story as a self-taught AI researcher
#54I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understand…
I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?
> Mind providing some concrete examples?
It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications.
To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.
Re: My story as a self-taught AI researcher
#55This reeks of survivorship bias to me. I much prefer Andreas Madsen's more sober and self-conscious take on independent research [0]. > I’d spend 1-2 months completing Fast.ai course V3, and spend another 4-5 months completing personal projects or participating in machine learning competitions... After six months, I’d recommend doing an internship. Then you’ll be ready to take a job in industry or do consulting to se…
> Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020?
I don't think the idea is to look for an internship after the course but an additional 4 months of personal projects. After applying state of the art deep learning for 4 months full time you'll have some very cool projects, and you could probably convince some company to take you on as an intern for a certain amount of time.
Re: My story as a self-taught AI researcher
#56Is this guy actually a researcher in the way most people would think of it? That is, someone who pushes the boundaries of science; who develops new AI techniques or finds the hard boundaries of existing AI techniques; who finds new ways compose multiple AI techniques cohesively; who explores the theoretical foundations of AI. Or is he someone who uses AI techniques to solve problems (and then wrote a paper about it)?…
Most people would be wrong if they think that this is what all researchers do
Re: My story as a self-taught AI researcher
#57Is this guy actually a researcher in the way most people would think of it? That is, someone who pushes the boundaries of science; who develops new AI techniques or finds the hard boundaries of existing AI techniques; who finds new ways compose multiple AI techniques cohesively; who explores the theoretical foundations of AI. Or is he someone who uses AI techniques to solve problems (and then wrote a paper about it)?…
For better or worse, the definition of researcher has morphed into a combination of 1. Solves previously unsolved problems 2. Publishes papers sharing those solutions without regard to the kind/spirit/scope of problems solved. Since conference publications don’t have the same number constraints as journal papers, and are accepting of application-specific results, this explosion of what is considered “research” is som…
[deleted]
Re: My story as a self-taught AI researcher
#58I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understand…
Is AlphaFold 1 or 2?
Re: My story as a self-taught AI researcher
#59Earlier quoted context omitted.
So if I understand correctly, to reformulate in my own words/views: while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal), fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is…
yep, you read that right. Source: I am a PhD student at Stanford at the Stanford Vision and Learning lab ( http://svl.stanford.edu/ ) and read a ton of AI papers. The vast majority of papers are done with datasets anyone can just download / request, as far as I've seen.
personally, without affiliation to a university, I have a hard time downloading the datasets through my slow home connection. I live in a city with a university, I explained the situation but they won't let me download a dataset even if I pay, ... only when I enroll. Instead of just selling the shovel, they want to sell me the wheelbarrow too.
I succeeded one time in convincing the guy behind a desk in an internet cafe, so I could bring my HDD and download a dataset in a calmer time of day, and throttled so it wouldn't disturb other customers. This went without any problems for the other customers in the internet cafe. When I asked again a few months later for a new dataset, they no longer wanted me to do so...
There seems to be no download by mail service (and I only get people forwarding me to google cloud products etc, which as a European is so financially out there with automatic balance deductions and non transparent pricing schemes, I would have no qualms using GCP or others if they ran a prepaid alternative for people who refuse to take on risk)
Re: My story as a self-taught AI researcher
#60This was a great read (and great nuggets, like that paper on Intelligence by Chollet). I wonder: — Is math a problem for non-academic researchers? Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but ma…
“ I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny“
Thanks to transfer learning, tiny datasets are not a major issue to developing AI solutions. Fast.ai makes it super easy to overcome that hurdle.