I've not kept track of where large transformers like this have gotten to, GPT3 and the like - has GPT3 made any real difference to the world? Are people using it? Has it vastly improved any software?
DeepMind’s new AI with a memory outperforms algorithms 25 times its size
51–60 of 141 posts
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#52Earlier quoted context omitted.
Searching for "child" or "house" will yield what has been classified as such in training - and searching for Japanese or Thai labels will do the same. No surprise there, if the labels don't get normalized before training. And normally, that's harmless - as you said, you'd expect to see an AI finding pictures of houses in the region/culture you are searching it. But in a multi-cultural/multi-ethnic society, searching…
No, remediating such issues (only predicting the maximum likelihood class in the dataset) is a problem of _machine learning and optimization_ research, not ethics research. There is nothing an ethicist can do to solve this problem. It is easy to point out problems with existing AI and write a bunch of papers to get yourself tenure. It is very hard to fundamentally advance our understanding of deep learning models pas…
> _machine learning and optimization_ research
Ethics education is (unfortunately) not really seen as necessary across the tech field, which is why ethics researchers need to be part of at all stages of AI development.
And for what it's worth, ethics researchers should be part of all technology development - the "racist soap dispenser" should have been more than enough proof of how even a very simple, innocent product can contribute to ethnic discrimination.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#53Sounds as if they stored all the correct answers in a database and call it "better". How do they even evaluate these models? Like they already have a billion preprepared correct answers in the database. How do they come up with new questions for the evaluation?
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#54Earlier quoted context omitted.
The problem is that AI (and the English language to some extent) transcends borders. So even if it's an AI developed in the US, it can potentially impact people outside the US and it makes ethical sense to build something that doesn't exclude groups based on arbitrary conditions.
Yes, but to offset that, many a.i. in English were also made outside of English-speaking regions, in what one assumes to be proportional degree. This is probably why there is more variance when searching for English terms as wel, as a Lingua Franca . If I search “house” I do see some styles of architecture not commonly found in Anglo-Saxon nations, whereas all occurrences of “huis” do seem to be situated in the Nethe…
> many a.i. in English were also made outside of English-speaking regions
Different regions, yes - but where did the training and benchmark datasets come from? AI research is surprisingly monocultural (or use "standardized benchmarks" if you're feeling charitable). Not too long ago, there was a paper posted on HN that showed that a bunch of the datasets contain mislabeled data, which means a lot of "different" models are encoding similar biases.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#55If we point it at the horrendously bad scots wiki (some kid in the US decided he'd translate Wikipedia into what he thought was lowland scots/Doric.. it's a disaster) we might get entertainingly bad outcomes.
Note, the stuff written by said kid has long since been deleted. However i have no idea what the quality of the rest of scowiki is.
Well, it is pretty hard to make something in a language when it is a dialect continuum and not a standardized variety that is forced onto the whole population through the education system and media.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#56Earlier quoted context omitted.
I got into a very long debate with an openai person 4/5 years ago about this + adversarial learning + access to a quantum computer (think just straight up world class abacus) was close to the primitives required for more generalized AI. They didn't agree with me, but that's ok! :)
Does anyone know if we understand enough about natural, generally intelligent brains to dismiss the idea that they are using quantum phenomenon for computation? Is it unlikely for any reason?
Roger Penrose’s The Emperors New Mind proposed this in the 1990s. He gave circumstantial possibilities, and I don’t think we know more than that now.
As an amateur, my instinct is that the mystery of the observer "causing" wave function collapse is the best clue we have either way.
I wouldn’t be surprised if we are just weighted neurons, or if we do something quantum too. I doubt the something quantum will be the same as what our quantum computers do.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#57Earlier quoted context omitted.
I got into a very long debate with an openai person 4/5 years ago about this + adversarial learning + access to a quantum computer (think just straight up world class abacus) was close to the primitives required for more generalized AI. They didn't agree with me, but that's ok! :)
Does anyone know if we understand enough about natural, generally intelligent brains to dismiss the idea that they are using quantum phenomenon for computation? Is it unlikely for any reason?
I think people bring up this quantum phenomenon stuff a bit too often in an attempt to add a layer of mysticism to humanity. We should only jump to that conclusion if we have some kind of proof or reason to believe so. We don’t even understand a lot of how the brain works at the macro scale yet so it’s a bit early to start saying it’s dependent on those processes (other than in the same way all physical systems are dependent on quantum phenomena.)
I’m also not a neuroscientist or a physicist though, so this is just my relative layman’s take.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#58Very interesting. GPT-J is an opensource free alternative to GPT-3 and requires at least 12.1GB memory to run the model (which is reduced from original 48GB ram). But if the model stores some kind of index and does internet searches (or hard drive) instead, then it could scale much further as there is a limit on how much memory you can use in production.
Just doing some napkin math, the whole GPT-J corpus was around 500 billion tokens, which at 4 tokens per byte would be roundabout 2 Terabyte. That, parked on a fast NVMe SSD will give you roundabout 1MM random lookups per second. Even with some transfers inbetween, this should be more than enough to not just perform in equal time, but probably less — as well as cost you less than the GPU you need for the reduced size model.
Exciting times.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#59> Gebru, a widely respected leader in AI ethics research, is known for coauthoring a groundbreaking paper that showed facial recognition to be less accurate at identifying women and people of color, which means its use can end up discriminating against them. Surely this is a function of location? I understand the U.S.-English term “person o color” to be convoluted language for “not white”. One simple thing I notice i…
Searching for "child" or "house" will yield what has been classified as such in training - and searching for Japanese or Thai labels will do the same. No surprise there, if the labels don't get normalized before training. And normally, that's harmless - as you said, you'd expect to see an AI finding pictures of houses in the region/culture you are searching it. But in a multi-cultural/multi-ethnic society, searching…
No post body was provided.
Re: DeepMind’s new AI with a memory outperforms algorithms 25 times its size
#60Earlier quoted context omitted.
Note, the stuff written by said kid has long since been deleted. However i have no idea what the quality of the rest of scowiki is.
Well, it is pretty hard to make something in a language when it is a dialect continuum and not a standardized variety that is forced onto the whole population through the education system and media.
Not really relevant to the topic in question but... Isn't this how most languages begin?
First you have a continuum of language dialects, then one of them dominates for political reasons, then it gets codified, then enforced onto everybody through centralised education. Dialects not under direct unified political control become related but separate languages... And so on.