Yann LeCun on GPT-3

> It couldn't possibly make the connection that river and streams are bodies of water, because there is no association with reality.

Yet if you asked it if rivers and streams are bodies of water it would probably say that they are.

Likewise if I asked you if black holes and neutron stars are both celestial bodies you would say yes... but you've presumably never seen them, only read about them.

Now I think you could argue that you could ultimately tie your knowledge back to some reality, like star ties to the sun and how you've seen the sun, but I haven't been so convinced that we know enough about how the mind works to be sure that the distinction between form and meaning is real.

Assuming there is no "soul" which makes meat life special, I don't see any fundamental problems with building simulated intelligence.

Actually - why would a bot be more likely to make typos and grammatical errors? Surely a slightly careless human is the simpler explanation?

I'd also suggest Wittgenstein's Tractatus Logico-Philosophicus, a seminal work of the logical positivist movement. Influenced by Frege's predicate calculus, the aim of the Tractatus was to determine an isomorphic relationship between language, thought, and external states of affairs. An axiomatic attempt to reveal a potentially ideal logical language, that is not interested in meaning per se, but merely an accurate reflection of the world. A closed system that essentially excludes non-falsifiable metaphysical question. Famously concluding with the instruction: "Whereof one cannot speak, thereof one must be silent." Part of Wittgenstein's project, even in its early aggressively logical form, was philosophy as a therapeutic. That is, the metaphysical questions concerning god, being, essence, and forms that had inspired thousands of years worth of fevered conversation, could be finally be quieted. That's not to say they couldn't be meditated on, but were not in the domain of his logical language, and so silence. Again, I think early Wittgenstein sometimes gets misinterpreted, "...therefore one cannot speak" does not, to me, mean that it can't be considered or one must forgo spirituality, just that it couldn't be spoken of within the project of the Tractatus.

Logical empiricism was ultimately a dead end as the criteria for even verifying empirical truth has long been contentious philosophically, and was further critiqued by contemporaries such as Quine who attacked the premise of the analytic/synthetic distinction (think Hume's fork, which Kant tried to solve) and Popper who cited the problem of induction to critique the fundamental premises of the positivists verificationism.

Wittgenstein is an interesting case, as the Tractatus is considered an early work of his, profoundly influential to analytic philosphy at the time, yet his later work, Philosophical Investigations is sometimes seen to retract the dogmatism found in the Tractatus. I tend to take the view that it's a continuation of his thought, rather than a retraction of his earlier work. Crudely, whereas his former thought represented a narrowly axiomatic definition of language and its truth value, PI investigates, among many other ideas, language as an activity, or game, that has meaning dependent on the context of its use, languages as families. Granted, Wittgenstein is a complex thinker and these are simply my interpretations.

It's also curious to note that as positivism was beginning to fall out of favor around the time of the second world war, a continental thinker such as Heidegger, whose thought luxuriated in the kind of metaphysical questions the positivists necessarily eschewed, rose to prominence and was infamously sanctioned by the NSDAP to philosophize about their presumed "destiny". Bit of a tangent, but I think the historical context is relevant, as often philosophical movements are birthed from pre- and post-war attitudes.

Text reproduced, minus abusive shell of dark patterns:

Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do.

This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare.

GPT-3 is a language model, which means that you feed it a text and ask it to predict the continuation of the text, one word at a time. GPT-3 doesn't have any knowledge of how the world actually works. It only appears to have some level of background knowledge, to the extent that this knowledge is present in the statistics of text. But this knowledge is very shallow and disconnected from the underlying reality.

As a question-answering system, GPT-3 is not very good. Other approaches that are explicitly built to represent massive amount of knowledge in "neural" associative memories are better at it.

As a dialog system, it's not very good either. Again, other approaches that are explicitly trained to perform to interact with people are better at it. It's entertaining, and perhaps mildly useful as a creative help. But trying to build intelligent machines by scaling up language models is like a high-altitude airplanes to go to the moon. You might beat altitude records, but going to the moon will require a completely different approach.

It's quite possible that some of the current approaches could be the basis of a good QA system for medical applicatioms. The system could be trained on the entire medical literature and answer questions from physicians.

But compiling massive amounts of operational knowledge from text is still very much a research topic.

Several things. Mostly that it has to be rolled out slowly because people lives are at stake so testing can run for quite a bit longer than you’d expect. Also that everyone wants to be the one who makes the breakthrough so companies will claim its right around the corner (like fusion) repetitively, i.e. Tesla saying it’d be here in 2018. We’re just at a point where these things can use parking lots so I wouldn’t expect a complete rollout several years as systems are built on top of other systems that have been widely tested and confirmed to work.