Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made a transformer to predict a simple sequence manually

vgel.me

Post reply on HN
Print view View original

81–90 of 104 posts

Re: I made a transformer to predict a simple sequence manually

#81

Post by KirillPanov »

ntonozzi wrote: post #63
PaulHoule wrote: post #12

It's some kind of abstract machine, like a turing machine or the machine that parses regexes, isn't it?

Yes! Check out this paper describing how Linear Transformers are secretly Fast Weight Programmers: https://arxiv.org/abs/2102.11174 .

Jürgen Schmidhuber reminds me of Richard Feynman in one very specific way: while everybody else in their respective fields uses math to hide the deep insights they've been mining for publications, Schmidhuber and Feynman just simply tell you the big insight, and then proceed to refine and illuminate it with math.

Re: I made a transformer to predict a simple sequence manually

#82

Post by PoignardAzur »

Supply5411 wrote: post #37

I've been kicking around a similar idea for awhile. Why can't we have an intuitive interface to the weights of a model, that a domain expert can tweak by hand to accelerate training? For example, in a vision model, they can increase the "orangeness" collection of weights when detecting traffic cones. That way, instead of requiring thousands/millions more examples to calibrate "orangeness" right, it's accelerated by a…

The reason you're looking for is called "The Bitter Lesson".

The short version is, trying to give human assistance to AIs is almost always less cost-effective than making them run on more computing power.

By the time your human expect has calibrated your weight layers to detect orange traffic cones, your GPU cluster has trained its AI to detect traffic cones, traffic ligts, trees, other cars, traffic cones with a slightly different shade of orange, etc.

Re: I made a transformer to predict a simple sequence manually

#85

Post by seethedeaduu »

GaggiX wrote: post #84
seethedeaduu wrote: post #83

Earlier quoted context omitted.

I don't think they undergo hormonal treatment just to transition their gender.

[flagged]

Gender-affirming care, also known as transition (transition what? Their sex!), is done in order to make trans people more comfortable in society, in their inner self (hormones affect mood, behavior, feelings, etc) and in their body by changing their sex. This is why they are prescribed sex hormones as these are what guides gene expression and decide sex-dependent characteristics. While some trans people would be content with just changing their physical appearance most of them seem to actually aim to change their sex.

Re: I made a transformer to predict a simple sequence manually

#86

Post by voop »

seethedeaduu wrote: post #85
GaggiX wrote: post #84

Earlier quoted context omitted.

[flagged]

Gender-affirming care, also known as transition (transition what? Their sex!), is done in order to make trans people more comfortable in society, in their inner self (hormones affect mood, behavior, feelings, etc) and in their body by changing their sex. This is why they are prescribed sex hormones as these are what guides gene expression and decide sex-dependent characteristics. While some trans people would be cont…

[dead]

Re: I made a transformer to predict a simple sequence manually

#87

Post by seethedeaduu »

voop wrote: post #86
seethedeaduu wrote: post #85

Earlier quoted context omitted.

Gender-affirming care, also known as transition (transition what? Their sex!), is done in order to make trans people more comfortable in society, in their inner self (hormones affect mood, behavior, feelings, etc) and in their body by changing their sex. This is why they are prescribed sex hormones as these are what guides gene expression and decide sex-dependent characteristics. While some trans people would be cont…

[dead]

We are not discussing whether sex is a spectrum or not though. Was the nonbinary person that you talked with undergoing medical transition? If not this might explain their view.

Re: I made a transformer to predict a simple sequence manually

#88

Post by civilized »

PoignardAzur wrote: post #82
Supply5411 wrote: post #37

I've been kicking around a similar idea for awhile. Why can't we have an intuitive interface to the weights of a model, that a domain expert can tweak by hand to accelerate training? For example, in a vision model, they can increase the "orangeness" collection of weights when detecting traffic cones. That way, instead of requiring thousands/millions more examples to calibrate "orangeness" right, it's accelerated by a…

The reason you're looking for is called "The Bitter Lesson". The short version is, trying to give human assistance to AIs is almost always less cost-effective than making them run on more computing power. By the time your human expect has calibrated your weight layers to detect orange traffic cones, your GPU cluster has trained its AI to detect traffic cones, traffic ligts, trees, other cars, traffic cones with a sli…

The AI community has over-learned this bitter lesson and is now unprepared to make useful AI in data-constrained environments.

This is why almost all of the countless enterprise AI demos taking place as we speak are doomed to failure.

Re: I made a transformer to predict a simple sequence manually

#89

Post by voop »

seethedeaduu wrote: post #87
voop wrote: post #86

Earlier quoted context omitted.

[dead]

We are not discussing whether sex is a spectrum or not though. Was the nonbinary person that you talked with undergoing medical transition? If not this might explain their view.

The wider point made in that piece is, it's impossible to change sex. For humans and other mammals at least.

Re: I made a transformer to predict a simple sequence manually

#90

Post by seethedeaduu »

voop wrote: post #89
seethedeaduu wrote: post #87

Earlier quoted context omitted.

We are not discussing whether sex is a spectrum or not though. Was the nonbinary person that you talked with undergoing medical transition? If not this might explain their view.

The wider point made in that piece is, it's impossible to change sex. For humans and other mammals at least.

Let's agree to disagree. Out of curiosity, did you adopt this belief before or after talking with your friend?
Post reply on HN