381–390 of 1001 posts
Is that the same confidence measure you can tease out by prompting "to each of your statements output your estimated confidence in it's truthfulness" ?
I think it's interesting that they've benchmarked it against an array of standardized tests. Seems like LLMs would be particularly well suited to this kind of test by virtue of it being simple prompt:response, but I have to say...those results are terrifying. Especially when considering the rate of improvement. bottom 10% to top 10% of LSAT in What are the implications for society when general thinking, reading, and…
What is more bizarre is that all of it's errors seem to be multiples of 60!
I'm wondering if it is confusing 60 based time (hour second) computations for regular multiplication?
Example:
xGPT 987 456 321
437 428919 199512 140397
654 645258 298224 209994
123 121401 56088 39483
x 987 456 321
437 431319 199272 140277
654 645498 298224 209934
123 121401 56088 39483
error 987 456 321
437 2400 -240 -120
654 240 0 -60
123 0 0 0Test taking will change. In the future I could see the student engaging in a conversation with an AI and the AI producing an evaluation. This conversation may be focused on a single subject, or more likely range over many fields and ideas. And may stretch out over months. Eventually teaching and scoring could also be integrated as the AI becomes a life-long tutor. Even in a future where human testing/learning is no l…
I think this will totally change the way we educate and test. As someone for whom the education system really didn't serve well, I am very excited.
A class of problem that GPT-4 appears to still really struggle with is variants of common puzzles. For example: >Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three…
From the paper: > Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar. I'm curious whether they have continued to scale up model size/compute significantly or if they have managed to make significant innovati…
I bet they use CLIP to caption the image and feed the text of the caption into GPT, but that's just a guess.
Folks who made this happen: https://openai.com/contributions/gpt-4
I will not be surprised if by the time GPT-5 releases, the paper and project will be completely anonymized.
I think it's interesting that they've benchmarked it against an array of standardized tests. Seems like LLMs would be particularly well suited to this kind of test by virtue of it being simple prompt:response, but I have to say...those results are terrifying. Especially when considering the rate of improvement. bottom 10% to top 10% of LSAT in What are the implications for society when general thinking, reading, and…
> Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar. "Open"