Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT scores 80% on SAT reading/writing with collective chain of thought

old.reddit.com

Post reply on HN
Print view View original

11–20 of 126 posts

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#11

Post by guiambros »

lossolo wrote: post #10
guiambros wrote: post #8

Earlier quoted context omitted.

Not so fast... https://www.learngpt.com/posts/the-room-has-19-chairs-we-are... Edit: even better: https://www.learngpt.com/posts/the-room-has-19-chairs-we-are...

And answer from the new version: > After using 10 chairs, we have 19 chairs - 10 chairs = 9 chairs in the room. Then after buying 9 more chairs, we have 9 chairs + 9 chairs = 18 chairs in the room.

Which version are you using? I'm on "ChatGPT Jan 9 Version", and not only it got right, but also why someone would (incorrectly) guess 18. See edit above.

But of course, there's an initial entropy to the models, so it's very possible that depending on the initial state it converged into the wrong (but yet probabilistically plausible) answer.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#12

Post by dymk »

nigamanth wrote: post #9

ChatGPT could easily get 100% on the Mathematics sections, but I never thought it'd get 80% on the reading/writing section.

I’d expect the opposite. It’s not very good at precise problem solving but it’s very good at writing verbose flowery bullshit.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#13

Post by charcircuit »

dymk wrote: post #12
nigamanth wrote: post #9

ChatGPT could easily get 100% on the Mathematics sections, but I never thought it'd get 80% on the reading/writing section.

I’d expect the opposite. It’s not very good at precise problem solving but it’s very good at writing verbose flowery bullshit.

You can have it call out to python or some other runtime to help with that. It just needs to be able to turn a problem into code.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#14

Post by zug_zug »

eternalban wrote: post #2

Using chain of thought [slightly modified] example from https://arxiv.org/abs/2201.11903 (ref'd in OP) Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11. Q: The room has 19 chairs. We are using 10 and then bought 9 mor…

| Q: The room has 19 chairs. We are using 10 and then bought 9 more, how many chairs do we have in the room?

I don't know if you made this question up, but I think this question has issues (I as a reader find it meaningless). You say you bought 9 more chairs, but you don't say you put those 9 chairs into the room. Also you don't specify "we" own any of the 19 chairs in the room initially, but at the end you ask how many of those chairs "we have."

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#16

Post by nigamanth »

dymk wrote: post #12
nigamanth wrote: post #9

ChatGPT could easily get 100% on the Mathematics sections, but I never thought it'd get 80% on the reading/writing section.

I’d expect the opposite. It’s not very good at precise problem solving but it’s very good at writing verbose flowery bullshit.

It can easily do mathematic functions and SAT syllabus. Despite the word problems, by assuming ChatGPT should easily be able to solve simple arithmetic.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#17

Post by texaslonghorn5 »

dymk wrote: post #12
nigamanth wrote: post #9

ChatGPT could easily get 100% on the Mathematics sections, but I never thought it'd get 80% on the reading/writing section.

I’d expect the opposite. It’s not very good at precise problem solving but it’s very good at writing verbose flowery bullshit.

"reading/writing" is actually reading comprehension/grammar. the essay was optional from [####] until june 2021 when it was discontinued. so the ai isn't writing, it's answering "writing" questions which are grammar/sentence structure/correctness/etc.

also if you click the link, it was all multiple choice.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#18

Post by scarecrw »

I've played around with this myself a fair bit, so I'll add two things:

* I'm not sure if I buy the "chain of thought" approach as being very significant here. I haven't recorded my results, but I've gotten ChatGPT to answer correctly a significant number of SAT questions with much more direct approaches (e.g. "The following is a multiple choice reading question, answer the question and explain why your answer is correct").

* This was likely significantly benefited by the new format of the reading/writing section of the SAT. Previously, a substantial number of questions relied on longer passages and connecting ideas between paragraphs. These questions are from the newest version of the test which is exclusively fill in the blank or short passage questions.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#19

Post by tshaddox »

rayiner wrote: post #6
sublinear wrote: post #4

I think this is evidence against the usefulness of standardized tests rather than support for ChatGPT

Why? This section is literally just questions about a modestly complex reading passage. It’s the most real world part of the SAT.

Also you’re not allowed to use any reference materials and certainly not the Internet when you’re taking these tests.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#20

Post by kaoD »

guiambros wrote: post #11
lossolo wrote: post #10

Earlier quoted context omitted.

And answer from the new version: > After using 10 chairs, we have 19 chairs - 10 chairs = 9 chairs in the room. Then after buying 9 more chairs, we have 9 chairs + 9 chairs = 18 chairs in the room.

Which version are you using? I'm on " ChatGPT Jan 9 Version ", and not only it got right, but also why someone would (incorrectly) guess 18. See edit above. But of course, there's an initial entropy to the models, so it's very possible that depending on the initial state it converged into the wrong (but yet probabilistically plausible) answer.

I just tried it. Jan 9 version. Got the same response as OP. I included the question about tennis balls first, wondering if that makes a difference.

EDIT: tried on a fresh conversation and it guessed correctly. Interesting.

Post reply on HN