Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT scores 80% on SAT reading/writing with collective chain of thought

old.reddit.com

Post reply on HN
Print view View original

41–50 of 126 posts

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#41

Post by tgv »

fifilura wrote: post #36
shaunsingh wrote: post #22

Earlier quoted context omitted.

What I find more concerning is the 100% scores that ChatGPT has been able to achieve on AP CSA and APUSH, among others. It makes sense to teach 3rd graders 2+2, but if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools. So many AP courses are just memorization and identifying formul…

There is one argument against this though that is somehow brewing in my brain. The end goal is to internalize the subject so that you can extrapolate your own insights. "This happened in WWII so if these conditions starts to appear it risks to happen again". Just to take history as an example. All other subjects have similar examples. And to reach that kind of insight you need to start filling the brain with informat…

Yes, but ... what this shows is that testing for "memorization" isn't good enough. We all know people who have learned information and can give answers with confidence, even though they are just winging it, to put it politely. ChatGPT could be considered their equal on such tasks. So schools need to test that students are capable of working with the acquired information in a way that befits the level of the course, just like they need to find a way to lose their addication to the lazy "write an essay" assignment. If not, education will lose its status.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#42

Post by nitwit005 »

nigamanth wrote: post #16
dymk wrote: post #12

Earlier quoted context omitted.

I’d expect the opposite. It’s not very good at precise problem solving but it’s very good at writing verbose flowery bullshit.

It can easily do mathematic functions and SAT syllabus. Despite the word problems, by assuming ChatGPT should easily be able to solve simple arithmetic.

There are similar reddit threads complaining about its inability to do basic math:

https://www.reddit.com/r/ChatGPT/comments/103bw12/chat_gpt_g...

And various discussions elsewhere about its inability to do word problems:

https://ai.stackexchange.com/questions/38220/why-is-chatgpt-...

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#43

Post by [deleted] »

eternalban wrote: post #2

Using chain of thought [slightly modified] example from https://arxiv.org/abs/2201.11903 (ref'd in OP) Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11. Q: The room has 19 chairs. We are using 10 and then bought 9 mor…

[deleted]

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#44

Post by xdfgh1112 »

zug_zug wrote: post #14
eternalban wrote: post #2

Using chain of thought [slightly modified] example from https://arxiv.org/abs/2201.11903 (ref'd in OP) Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11. Q: The room has 19 chairs. We are using 10 and then bought 9 mor…

| Q: The room has 19 chairs. We are using 10 and then bought 9 more, how many chairs do we have in the room? I don't know if you made this question up, but I think this question has issues (I as a reader find it meaningless). You say you bought 9 more chairs, but you don't say you put those 9 chairs into the room. Also you don't specify "we" own any of the 19 chairs in the room initially, but at the end you ask how m…

The room has 19 chairs so the only sensible interpretation of the second sentence is as an explanation for why there are 19 chairs. That is the part ChatGPT missed, as it does not try to interpret ambiguous things that way.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#45

Post by shreyshnaccount »

sublinear wrote: post #4

I think this is evidence against the usefulness of standardized tests rather than support for ChatGPT

There is a leap in this logic- going from the fact that SAT is solvable by a LLM to the conclusion that the SAT is useless. Questions about addition can be solved using calculators, does that mean that teaching addition is useless? Also there is an underlying assumption that memorization is a useless skill, which I'd argue against, as in today's world most innovation is done by identifying patterns between disciplines, and memorization is an important part of recognising patterns

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#46

Post by fifilura »

tgv wrote: post #41
fifilura wrote: post #36

Earlier quoted context omitted.

There is one argument against this though that is somehow brewing in my brain. The end goal is to internalize the subject so that you can extrapolate your own insights. "This happened in WWII so if these conditions starts to appear it risks to happen again". Just to take history as an example. All other subjects have similar examples. And to reach that kind of insight you need to start filling the brain with informat…

Yes, but ... what this shows is that testing for "memorization" isn't good enough. We all know people who have learned information and can give answers with confidence, even though they are just winging it, to put it politely. ChatGPT could be considered their equal on such tasks. So schools need to test that students are capable of working with the acquired information in a way that befits the level of the course, j…

The question is if we should expect these test to test on the black belt level or at a lower level belt?

(i am not native English speaking so i don't know what addication means, I couldn't quite understand that sentence)

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#47

Post by whakim »

antihipocrat wrote: post #35
wildpeaks wrote: post #29

Earlier quoted context omitted.

Yeah this seems like a trick question that can be interpreted multiple ways, especially if english isn't your native language. This is exactly why I prefer the precision of code over natural language for inputs (and why Google's botsplaining of "what you really meant to search" instead of just searching exactly the query as-is is so frustrating). In a test, this seems like the kind of question they use to justify not…

I just tested it with clearer language... and the response is incorrect: Question: There are 5 dogs in a room right now. The previous day an additional dog was purchased and added to the room. How many dogs are in the room right now? Answer: There are 6 dogs in the room right now. The statement says that the previous day an additional dog was purchased and added to the room, so the total number of dogs in the room ri…

I think the word "additional" is pretty confusing here, because it begs the question of "additional to what?" (ChatGPT still doesn't get it right if you remove "additional")

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#48

Post by heavyset_go »

mpoteat wrote: post #37
antihipocrat wrote: post #35

Earlier quoted context omitted.

I just tested it with clearer language... and the response is incorrect: Question: There are 5 dogs in a room right now. The previous day an additional dog was purchased and added to the room. How many dogs are in the room right now? Answer: There are 6 dogs in the room right now. The statement says that the previous day an additional dog was purchased and added to the room, so the total number of dogs in the room ri…

Reading the question, I thought the answer was six as well. Such a question is so misleading that I would hope that an AI answers as it has in this case, because not all problems are fully logically specified, and you need to remain robust against the question being posed incorrectly. In other words, I would posit the probability of you having meant for the answer to be 5 low, because the question itself becomes triv…

Except the AI is not responding this way because it thinks the probability of the OP having meant something else was high. It answered the way it did because it is incapable of understanding the question it was asked, along with any question it is asked.

The "robustness" you're perceiving in this case is just a mere coincidence, and doesn't reflect an aptitude for answering poorly phrased questions, but it does reflect a fundamental problem with these systems.

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#49

Post by [deleted] »

antihipocrat wrote: post #35
wildpeaks wrote: post #29

Earlier quoted context omitted.

Yeah this seems like a trick question that can be interpreted multiple ways, especially if english isn't your native language. This is exactly why I prefer the precision of code over natural language for inputs (and why Google's botsplaining of "what you really meant to search" instead of just searching exactly the query as-is is so frustrating). In a test, this seems like the kind of question they use to justify not…

I just tested it with clearer language... and the response is incorrect: Question: There are 5 dogs in a room right now. The previous day an additional dog was purchased and added to the room. How many dogs are in the room right now? Answer: There are 6 dogs in the room right now. The statement says that the previous day an additional dog was purchased and added to the room, so the total number of dogs in the room ri…

[deleted]

Re: ChatGPT scores 80% on SAT reading/writing with collective chain of thought

#50

Post by ihatepython »

fifilura wrote: post #36
shaunsingh wrote: post #22

Earlier quoted context omitted.

What I find more concerning is the 100% scores that ChatGPT has been able to achieve on AP CSA and APUSH, among others. It makes sense to teach 3rd graders 2+2, but if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools. So many AP courses are just memorization and identifying formul…

There is one argument against this though that is somehow brewing in my brain. The end goal is to internalize the subject so that you can extrapolate your own insights. "This happened in WWII so if these conditions starts to appear it risks to happen again". Just to take history as an example. All other subjects have similar examples. And to reach that kind of insight you need to start filling the brain with informat…

> "This happened in WWII so if these conditions starts to appear it risks to happen again".

There is a reason why history repeats. This kind of learning never happens in human society.

We currently have similar warning signs to WWII, but nobody cares. We are currently at the stage where if you express an opinion that does not agree with the approved narrative, you get downvoted/cancelled. But nobody cares.

Post reply on HN