My story as a self-taught AI researcher • Hacker News Board

Re: My story as a self-taught AI researcher

#21

Post by jessaustin » Mon, Jan 20, 2020, 8:47 PM UTC

Nasrudith wrote: post #7

qntty wrote: post #6

"Many are realizing that education is a zero-sum credential game." Can this silly meme die already? Maybe it's understandable coming from an economist who values education for no other reason than it's economic effects, but it's strange coming from someone who clearly understands the value of personal development.

It is pretty strange even from an economist really - they of all people should be able to understand and articulate the difference between signaling value and direct utility value of a given good or service.

If anyone could realize the tenuous value of education, it might be someone paying student loans for an economics degree...

Re: My story as a self-taught AI researcher

#22

Post by eanzenberg » Mon, Jan 20, 2020, 8:50 PM UTC

TrainedMonkey wrote: post #5

woah wrote: post #3

Earlier quoted context omitted.

On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques

Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.

humans dont start with a random scrambled brain

Re: My story as a self-taught AI researcher

#23

Post by ummonk » Mon, Jan 20, 2020, 8:53 PM UTC

TrainedMonkey wrote: post #5

woah wrote: post #3

Earlier quoted context omitted.

On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques

Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.

Transfer learning for the win.

Re: My story as a self-taught AI researcher

#24

Post by AlanSE » Mon, Jan 20, 2020, 8:55 PM UTC

cygaril wrote: post #15

TrainedMonkey wrote: post #5

Earlier quoted context omitted.

Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.

Or our entire evolutionary history of data.

...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.

Re: My story as a self-taught AI researcher

#25

Post by K0SM0S » Mon, Jan 20, 2020, 8:56 PM UTC

cygaril wrote: post #15

TrainedMonkey wrote: post #5

Earlier quoted context omitted.

Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.

Or our entire evolutionary history of data.

It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us.

The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself.

It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).

Re: My story as a self-taught AI researcher

#26

Post by why-el » Mon, Jan 20, 2020, 9:01 PM UTC

codebolt wrote: post #19

qntty wrote: post #6

"Many are realizing that education is a zero-sum credential game." Can this silly meme die already? Maybe it's understandable coming from an economist who values education for no other reason than it's economic effects, but it's strange coming from someone who clearly understands the value of personal development.

My prediction is that whoever comes up with the next forward leap in AI will be someone who at minimum has a firm grasp on the various branches of undergraduate level maths. Naively tinkering with heuristic statistical ML methods like neural nets and hoping that higher level intelligence somehow magically pops out isn't the way forward. We need a more sophisticated approach.

This is already being done in places such as the university of Arizona (Chomsky and his former students). The subject is narrower of course (computational linguistics and some neuroscience), but there are taking an approach that is more Galilean in nature, by designing experiments that reduce externalities rather that simply looking at massive amounts of data. I think that's what's going be the most useful, at least in areas that continue to be challenging for the current trends in AI, namely language.

Re: My story as a self-taught AI researcher

#27

Post by mendeza » Mon, Jan 20, 2020, 9:10 PM UTC

woah wrote: post #3

narenst wrote: post #2

This is a really good time to be a Independent Scientist (aka Gentleman scientist) in this field because how nascent deep learning and similar techniques are. It requires a lot of trial and error and time/cost investment to bring the AI techniques to the masses. The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new…

On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques

I think an exciting area that can innovate the lack of data is domain randomization, and synthetic data generation.

Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reali...

http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T...

And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText

Re: My story as a self-taught AI researcher

#28

Post by tnecniv » Mon, Jan 20, 2020, 9:11 PM UTC

andreyk wrote: post #18

narenst wrote: post #2

This is a really good time to be a Independent Scientist (aka Gentleman scientist) in this field because how nascent deep learning and similar techniques are. It requires a lot of trial and error and time/cost investment to bring the AI techniques to the masses. The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new…

wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.

A lot of research data sets are publicly available, but many researchers based at universities have relationships with private companies where they can get access to data or other resources useful for research (e.g. Google has a big room of robotic arms generating data for pick and place tasks).

There is still plenty you can do with a reasonable personal budget, however.

Re: My story as a self-taught AI researcher

#29

Post by K0SM0S » Mon, Jan 20, 2020, 9:22 PM UTC

This was a great read (and great nuggets, like that paper on Intelligence by Chollet).

I wonder:

— Is math a problem for non-academic researchers?

Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but math nonetheless.

Not that you can't learn these on your own, but how big is the gap in practice, on the job, compared with actual PhDs in ML/math? (how much of a hinderance, a problem it is for the self-taught researcher)

— "Contracting" in the field of AI sounds great but, how exactly? Especially solo: what type of clients and how/where to find them, what type of 'business proposition' as a freelancer do you offer, what's the pricing structure of such gigs?

I mean, I can sell you websites and visuals and stuff, but AI? I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny and demand scripting skills to sort out (extract business value), not AI, so the value proposition is low for both parties; it's still early adoption so 90% don't even consider spending 1 cent on "AI" unless as a SaaS (they actually don't need to know if it's AI or programming).

I can imagine tons of fantastic research to do with SMBs, as partners or 'interested sponsors' (should they reap benefits on a low investment), but really not much yet in the way of "freelancer products" to market and sell for a living. I'm eagerly anticipating those days, but it's more like 2025-2030 as I see it.

I would love to hear first hand takes on this.

Re: My story as a self-taught AI researcher

#30

Post by yorwba » Mon, Jan 20, 2020, 9:26 PM UTC

K0SM0S wrote: post #20

andreyk wrote: post #18

Earlier quoted context omitted.

wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.

So if I understand correctly, to reformulate in my own words/views: while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal), fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is…

It's not exactly true that research institutions don't have access to the same big datasets as companies. For example, I took a course that involved tracking soccer players using videos provided by a streaming company that specializes in amateur soccer. They promised to give us access to their internal API under an NDA, which they wouldn't have done for just anyone.

On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.