Earlier quoted context omitted.
Yes, this stuff is fun to work with and use as an aid to programming. But I think what's dawning on a lot of us is that it seems to be able to facilitate a large increase productivity. And given that large productivity increase will industry need as many software developers (and other knowledge workers)?
Your opinion on this probably depends on what your job currently looks like. Everywhere I've worked there's been significantly more stuff we wanted to do than we could actually do. Speeding that up looks awesome to me. But if you work at BigCo and mostly sit on your hands already, then, yeah. I don't know.
Unpredictable abilities emerging from large AI models
221–230 of 326 posts
Re: Unpredictable abilities emerging from large AI models
#222Earlier quoted context omitted.
Indeed, it's fascinating to witness a sizable segment of the HN community and distinguished intellectuals like Chomsky displaying absolute denial. I've started bookmarking the comments so I can look back at them in 5 years and have a good laugh. Some from a few months back are already aging badly[0]. [0] https://news.ycombinator.com/item?id=34197033
Has Chomsky been right about anything?
Re: Unpredictable abilities emerging from large AI models
#223Earlier quoted context omitted.
Has Chomsky been right about anything?
My first introduction to him was when one uncontacted tribe or anothers' language refuted something he thought was fundamental to humans. In the decade(s) since I kinda think "Chomsky was wrong" is pretty much the only reason people bring him up.
When Chomsky proposed generative grammars, his theory of universal language acquisition, and so on, they were radical ideas that upturned the central cannon of linguistics.
Time has been on his side - entire schools and subfields of linguistics went extinct as more evidence emerged that Chomsky was fundamentally right. Basically every computer language and data format in existence is parsed/lexed in ways inspired by his models of language.
But now Chomsky is considered the stodgy old establishment, and whenever one of his theories is contradicted somewhere on the margins people shout "Aha! He was wrong the whole time!" and ignore the 99% of cases where his models are still the best ones we have.
Re: Unpredictable abilities emerging from large AI models
#224Earlier quoted context omitted.
The reason they seem to make things up is because they have no way to verify anything, they can only speak of things in relation to other things, but they have no epistemic framework. This is very much a fixable problem that augmentation with logic engines and a way to prioritise truth-claims could go some ways towards solving.
My memory could be improved by connecting my brain to an external hard drive. Wiring them together, alas, is not just hard; we have absolutely no idea how.
Re: Unpredictable abilities emerging from large AI models
#225I'd like to see posts on LLMs written from a different perspective. For me, the surprise comes not from the sudden emergent capability of language models, but that the understanding (and synthesis!) of ideas encoded in language has succumbed to literally nothing more than statistical analysis. Or at least come that much closer to doing so. That it bears so close a resemblance to actual thinking says more about the im…
I find the phrase "statistical analysis" a frustrating one nowadays as it seems to have become a signal for "I hold a particular philosophy of the mind". I don't understand this use of "statistical" as a diminutive to describe these models. Why can't incredibly complicated behavior be emergent from matrix multiplication subject to optimization in the same way that our biological matter has developed complicated emerg…
You're the one supposing a thing, so the burden of proof is on you. You need to demonstrate that the "incredibly complicated behaviour" that you're referring to (I assume this is longhand for "thinking", but please correct me if I'm wrong) is indeed emerging from matrix multiplication. Especially given that what you're suggesting is unexpected, given the known way these models work and the explanations that have been put forth already that extrapolate from the known way these models work.
If science were so credulous as to accept the first proffered theory about a new development, well, we wouldn't have these interesting AI models in the first place!
Re: Unpredictable abilities emerging from large AI models
#226Earlier quoted context omitted.
Give it 2 years, it will happen, and it will make billions. In 5 years, unlimited interactive NSFW video games that will be personalized and remember you.
In 10 years this could effectively be the end of Hollywood. Imagine if it can write a story and feed into 3d engine and output pitch perfect videos.
Re: Unpredictable abilities emerging from large AI models
#227Earlier quoted context omitted.
My first introduction to him was when one uncontacted tribe or anothers' language refuted something he thought was fundamental to humans. In the decade(s) since I kinda think "Chomsky was wrong" is pretty much the only reason people bring him up.
I see a lot of irony in this. When Chomsky proposed generative grammars, his theory of universal language acquisition, and so on, they were radical ideas that upturned the central cannon of linguistics. Time has been on his side - entire schools and subfields of linguistics went extinct as more evidence emerged that Chomsky was fundamentally right. Basically every computer language and data format in existence is par…
Re: Unpredictable abilities emerging from large AI models
#228Earlier quoted context omitted.
Can you elaborate on fundamental limits?
I’m guessing one is data. The limit would be once you’ve trained a LLM on all public (or even private) data. Sure you can still make some improvements or try to find some additional private data but still, a fundamental limit has been reached.
We don’t teach babies by throwing lots of data at them, instead we teach them by giving them useful data.
The loop I see is: - train on a lot of existing data - run out of useful data - people use ai and give feedback (we’re here) - perform reinforcement learning on the data collected
Loop over the last two steps.
There is already more than enough data available, it’s just not nicely labeled to say if it’s high quality or should be discarded. Those last two steps will implicitly do the labeling.
Re: Unpredictable abilities emerging from large AI models
#229Earlier quoted context omitted.
It's probably going to struggle with things it hasn't seen before?
> It's probably going to struggle with things it hasn't seen before? It wont. It'll just lie through its teeth and produce a very nice, very believable story which will unfortunately shatter when confronted with the real world.
There is a time and place for a tool. It's like watching someone make bad google queries then complain that they get shit results.
Re: Unpredictable abilities emerging from large AI models
#230Earlier quoted context omitted.
This is shortsighted. AI slave labor is the only route to a true overproduction and marxism, a welfare state. Or every body become slaves.
Who needs slaves? What use will they be? Why overproduce?