Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interview with gwern

dwarkeshpatel.com

Post reply on HN
Print view View original

181–190 of 262 posts

Re: Interview with gwern

#181

Post by aubanel »

nocobot wrote: post #95
aubanel wrote: post #90

Earlier quoted context omitted.

> For me that's just a great example of how basically the vast majority of AI influencers (who vie for influence on social media, rather than research) are basically clueless about AI and CS This is a bit stark: there are many great knowledgeable engineers and scientists who would not get your point about a^nb^n. It's impossible to know 100% of of such a wide area as "AI and CS".

is it really? this is the most common example for context free languages and something most first year CS students will be familiar with. totally agree that you can be a great engineer and not be familiar with it, but seems weird for an expert in the field to confidently make wrong statements about this.

In my country (France), I think most last-year CS students will not have heard of it (pls anyone correct me if I'm wrong).

Re: Interview with gwern

#183

Post by GistNoesis »

YeGoblynQueenne wrote: post #123
GistNoesis wrote: post #120

Earlier quoted context omitted.

a^nb^n can definitely be expressed and recognized with a transformer. A transformer (with relative invariant positional embedding) has full context so can see the whole sequence. It just has to count and compare. To convince yourself, construct the weights manually. First layer : zeros the character which are equal to the previous character. Second layer : Build a feature to detect and extract the position embedding…

>> The paper doesn't distinguish between what is the expressive capability of the model, and the finding the optimum of the model, aka the training procedure. They do. That's the whole point of the paper: you can set a bunch of weights manually like you suggest, but can you learn them instead; and how? See the Introduction. They make it very clear that they are investigating whether certain concepts can be learned by…

Gradient descent usually get stuck in local minimum, it depends on the shape of the energy landscape, that's expected behavior.

The current wisdom is that by optimizing for multiple tasks simultaneously, it makes the energy landscape smoother. One task allow to discover features which can be used to solve other tasks.

Useful features that are used by many tasks can more easily emerge from the sea of useless features. If you don't have sufficiently many distinct tasks the signal doesn't get above the noise and is much harder to observe.

That the whole point of "Generalist" intelligence in the scaling hypothesis.

For problems where you can write a solution manually you can also help the training procedure by regularising your problem by adding the auxiliary task of predicting some custom feature. Alternatively you can "Generatively Pretrain" to obtain useful feature, replacing custom loss function by custom data.

The paper is a useful characterisation of the energy landscape of various formal tasks in isolation, but doesn't investigate the more general simpler problem that occur in practice.

Re: Interview with gwern

#184

Post by adiabatty »

Vecr wrote: post #54
throw05678931 wrote: post #22

Lots of the claims about Gwern in the intro are exaggerated. Gwern is an effective altruist and his influence is largely limited to that community. It would be an exaggeration to claim that he influenced the mainstream of AI and ML researchers -- certainly Hinton, LeCun, Ng, Bengio didn't need him to do their work. He influences the AI safety crowd, who have ironically been trying to build AGI to test their AI safety…

"Wordcel" is roon terminology, right? I highly doubt Aschenbrenner is an EA, and if he's a "wordcel" he somehow managed to do mathematical economics without too much problem. Gwern's probably not a "wordcel" either, he can program, right? I've never seen any of his publications though. It's called Situational Awareness too, not "Situation Report", and Yudkowsky said he didn't like it. Not that Yudkowsky is an EA eith…

> Gwern's probably not a "wordcel" either, he can program, right? I've never seen any of his publications though.

He does Haskell stuff (he mentions it on his website), but he’s even better at words than he is as a mid-tier (I guess) Haskell programmer.

Whether this counts as a “wordcel” is an exercise left to the reader.

Re: Interview with gwern

#185

Post by [deleted] »

A_D_E_P_T wrote: post #10

www.gwern.net Something I've noticed in spending time online is that there's a "core group" of a few dozen people who seem to turn up everywhere there are interesting discussions. Gwern (who also posts here) is probably at the top of that list.

[deleted]

Re: Interview with gwern

#186

Post by selfhoster11 »

A_D_E_P_T wrote: post #10

www.gwern.net Something I've noticed in spending time online is that there's a "core group" of a few dozen people who seem to turn up everywhere there are interesting discussions. Gwern (who also posts here) is probably at the top of that list.

A thread comment from TeMPOraL is always a nice surprise.

Re: Interview with gwern

#187

Post by Vecr »

adiabatty wrote: post #184
Vecr wrote: post #54

Earlier quoted context omitted.

"Wordcel" is roon terminology, right? I highly doubt Aschenbrenner is an EA, and if he's a "wordcel" he somehow managed to do mathematical economics without too much problem. Gwern's probably not a "wordcel" either, he can program, right? I've never seen any of his publications though. It's called Situational Awareness too, not "Situation Report", and Yudkowsky said he didn't like it. Not that Yudkowsky is an EA eith…

> Gwern's probably not a "wordcel" either, he can program, right? I've never seen any of his publications though. He does Haskell stuff (he mentions it on his website), but he’s even better at words than he is as a mid-tier (I guess) Haskell programmer. Whether this counts as a “wordcel” is an exercise left to the reader.

You're a "wordcel" if you can't program and you can't do math. E.g. Scott Alexander. I don't care what roon says.

Re: Interview with gwern

#188

Post by selfhoster11 »

Matticus_Rex wrote: post #28
nprateem wrote: post #11

How we're using confirmation bias to credit one out of 8 billion people with special skills.

New to the deeper internet? Just about anyone who has been aware of gwern for long comes to believe he's got some special skills, even if they disagree with him. He's a niche celebrity entirely for being insightful and interesting.

If nothing else, the tech used for Gwern's website is positively inspiring for knowledge organisation system geeks. It's really cool and unique.

Re: Interview with gwern

#189

Post by Quinzel »

endtime wrote: post #162
Quinzel wrote: post #9

I don’t believe Gwern lives as frugally as he’s described in this (if this even actually is the real Gwern). I’m 100% sure that he has a persona he likes to portray and being perceived as frugal is a part of that persona. When it comes to answering the question “who is gwern?” I reckon Gwern’s a plant a seed in people’s mind type of guy, and let them come up with the rest of the story. Still, I like a lot of his writ…

I met Gwern once, when he came to the NYC Less Wrong meetup. I don't think he was internet-famous yet. It was probably 12 years ago or so, but based on my recollection, I'm totally willing to believe that he lives very frugally. FWIW he wouldn't have looked out of place at an anime convention.

What did he look like?

Re: Interview with gwern

#190

Post by nuz »

Quinzel wrote: post #9

I don’t believe Gwern lives as frugally as he’s described in this (if this even actually is the real Gwern). I’m 100% sure that he has a persona he likes to portray and being perceived as frugal is a part of that persona. When it comes to answering the question “who is gwern?” I reckon Gwern’s a plant a seed in people’s mind type of guy, and let them come up with the rest of the story. Still, I like a lot of his writ…

I believe him.
Post reply on HN