121–127 of 127 posts
The closer we get to artificial intelligence, the more we raise the bar for what qualifies as AI (as we should). Gopher/GPT-3 are already much more accurate than the average human at technical information retrieval (trivial to see from the dialogue transcripts: how many Americans know what a Schwarzschild metric is?). The focus on ethics and equity for these algorithms is interesting too, as the average human holds m…
It should have some uncertainty when it says there are no French-speaking countries in South America. French Guiana is there, but it's not clear it counts as a "country in South America" since it's part of France. Technically you could say France is (partially) a country in South America, and France definitely is French-speaking. The way the question is phrased is unclear as to whether French Guiana should count, and…
This model like its predecessor still lacks the ability to infer within constraints of physical world [1]. It does not have any notion of what a physical world is, and still is, at the end of the day, a statistical model. Attempting to do recreate the entirety of the physical world by describing it in text is an impressive feat, but a futile attempt. It is prone to inference errors that the model itself is not aware…
This is exactly what a language model does though, just at a different level of abstraction. It gives you a probability distribution over tokens at each step. That distribution can be narrow (low entropy, certain) or wide (high entropy, uncertain). The language output you see is just a sampling at some temperature from these distributions.
Though glancing at your paper I assume you are aware of this and I am missing the point you are making?
This model like its predecessor still lacks the ability to infer within constraints of physical world [1]. It does not have any notion of what a physical world is, and still is, at the end of the day, a statistical model. Attempting to do recreate the entirety of the physical world by describing it in text is an impressive feat, but a futile attempt. It is prone to inference errors that the model itself is not aware…
> 'I do not know' or 'I am not sure' for every occasion when it is not 100% sure in something (like a human can), this would drastically improve the usefulness. This is exactly what a language model does though, just at a different level of abstraction. It gives you a probability distribution over tokens at each step. That distribution can be narrow (low entropy, certain) or wide (high entropy, uncertain). The langua…
When a human does not know something it can tell that with 100% certainty.
Earlier quoted context omitted.
> 'I do not know' or 'I am not sure' for every occasion when it is not 100% sure in something (like a human can), this would drastically improve the usefulness. This is exactly what a language model does though, just at a different level of abstraction. It gives you a probability distribution over tokens at each step. That distribution can be narrow (low entropy, certain) or wide (high entropy, uncertain). The langua…
Statistical approaches require you to most commonly use a threshold. Sometimes the model output can be above the threshold and still wrong, and below the threshold and correct. You can never tell for sure, but just try to improve the benchmark average. This is not acceptable in most use cases where the wrong outcome of a single output can be disastrous. When a human does not know something it can tell that with 100%…
That problem aside, language models like Gopher are in fact generative, so no such threshold is needed! You instead sample from the implicit distribution.
Earlier quoted context omitted.
Statistical approaches require you to most commonly use a threshold. Sometimes the model output can be above the threshold and still wrong, and below the threshold and correct. You can never tell for sure, but just try to improve the benchmark average. This is not acceptable in most use cases where the wrong outcome of a single output can be disastrous. When a human does not know something it can tell that with 100%…
I don't see the difference between a human and a statistical model here. Surely in order to select an action to take, a person also has to apply some sort to threshold on their confidence? E.g. how is a doctor deciding to amputate or not amputate an organ based on an x-ray different from a classification model for the same task? That problem aside, language models like Gopher are in fact generative, so no such thresh…