Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I quiz ChatGPT about math

billwadge.com

Post reply on HN
Print view View original

41–50 of 96 posts

Re: I quiz ChatGPT about math

#41

Post by satvikpendem »

dxbydt wrote: post #32
HDThoreaun wrote: post #17

Earlier quoted context omitted.

You should basically be thinking of chatGPT as a personal Wikipedia. It has the collective knowledge of the internet. It can still be wrong about fact based questions, but is a good starting point. It’s not effective at doing math because that’s not what it was designed to do.

If I post that same question on Wikipedia and confidently assert the answer is 1, some good Samaritan will edit it within a day & say no dumbass its 6, and then link me to the Talk page & we'll duke it out over there. On ChatGPT, you have no recourse other than the thumb down button. When you give it the right answer it simply refuses to accept! It's sure amusing as an adult, but be aware that several schools (atleas…

You're misunderstanding the parent. They mean that in school, teachers tell you to use Wikipedia as simply a starting point rather than an authoritative source (indeed, look at the article's actual sources and references at the bottom and read those), not to literally use Wikipedia to perform mathematical operations.

In the same vein, use ChatGPT but verify its answers.

Re: I quiz ChatGPT about math

#42

Post by visarga »

rvz wrote: post #14

Anyone who has bothered to cut through the hype of ChatGPT, would already have realised that it isn't smart at math and is only regurgitating the results that it has been already been trained on and will present it to you to make itself 'appear' intelligent. It doesn't even know why it is gives you the wrong answer even when it is presenting the generated output as correct. Like what another commenter said, it has li…

GPT can be the generative (imagination) part of a system. The second part would be a system to validate. This means you can filter out the junk and keep the good samples, extending the training set. This is how the model can learn by performing massive search for solutions all on its own. A bit like AlphaGo.

So you might want to reevaluate it. It can sample really well, and it is hard to sample coherently in this combinatorial space. Sampling is half the task, validation needs more work.

Re: I quiz ChatGPT about math

#43

Post by panic »

HDThoreaun wrote: post #17
dxbydt wrote: post #12

Here's a very easy middle school competition problem - https://imgur.com/a/YdFSBVY It repeatedly got the answer wrong. I engaged in a long conversation with it (too long to post) where I tried to convince it that its answer was wrong, but it became quite rude at a certain point & alleged I was trying to trick it or something (?!) so I just gave up. Unfortunately, I was doing the whole exercise in front of my 10 year…

You should basically be thinking of chatGPT as a personal Wikipedia. It has the collective knowledge of the internet. It can still be wrong about fact based questions, but is a good starting point. It’s not effective at doing math because that’s not what it was designed to do.

ChatGPT, the free encyclopedia that no one can edit!

Re: I quiz ChatGPT about math

#44

Post by BoppreH »

kevinventullo wrote: post #23
SamBam wrote: post #15

I was actually very impressed with some math questions it was able to answer for me: S: If flabb means three, snabb means four, and whabb mean five, what is flabb plus whabb? If flabb means three, snabb means four, and whabb means five, then flabb plus whabb would be equal to three plus five, or eight. In this made-up numbering system, the words “flabb,” “snabb,” and “whabb” represent the numbers three, four, and fiv…

That is very cool, and surprising to me given I was also initially underwhelmed. One nitpick though: in the second question, although the explanation itself is clean, there appear to be seven b’s in its representation of the number six.

I've heard it's because GPT's input is whole words/tokens, not characters. So it has little insight into their spelling or letter positions, unless it's explicitly mentioned in the training set (e.g. rhymes are easy, counting letters of rare words is hard).

Though this is a specially bad example, surely it had bbbbbb somewhere in it's training set associated with "6".

Re: I quiz ChatGPT about math

#45

Post by copperx »

padjo wrote: post #29

If ChatGPT is what we can expect from AI I’m quite afraid. It seems to be usually correct but often spectacularly and confidently wrong. I hope we never give this technology decision making power.

Adding a confidence score to every answer doesn't look like an insurmountable problem.

Re: I quiz ChatGPT about math

#46

Post by CrazyStat »

dnissley wrote: post #18
CrazyStat wrote: post #6

Earlier quoted context omitted.

Prompt: > is 1023^2 even or odd? Response: > 1023 squared is even because it is equal to 1,046,529, which is an even number. To determine if a number is even or odd, you can check whether it is divisible by 2. If it is divisible by 2, it is even. If it is not divisible by 2, it is odd. > In general, any number that is a perfect square will be even if the number being squared is odd, and it will be odd if the number b…

Interesting, I got: > 1023 squared is 1048129, which is an odd number. To determine whether a number is even or odd, you can check whether it is divisible by 2. If it is, it is an even number. If it is not, it is an odd number. In this case, 1048129 is not divisible by 2, so it is an odd number.

Yeah, like I said it's about 50/50 whether it spits out even or odd. Try it a few times and you'll see.

Re: I quiz ChatGPT about math

#47

Post by LegitShady »

wyldfire wrote: post #34
SamBam wrote: post #15

I was actually very impressed with some math questions it was able to answer for me: S: If flabb means three, snabb means four, and whabb mean five, what is flabb plus whabb? If flabb means three, snabb means four, and whabb means five, then flabb plus whabb would be equal to three plus five, or eight. In this made-up numbering system, the words “flabb,” “snabb,” and “whabb” represent the numbers three, four, and fiv…

> There are a few reasons why people may argue about the answer to a mathematical question like “8 / 2(2+2)”. One reason is that there is more than one wav to interpret the question and arrive at a different answer. For example, some people may interpret the question as “8 / (2 * (2+2))” and arrive at an answer of 1, while others mav interpret it as “(8 / 2) * (2+2)” and arrive at an answer of 16. Wow, this is kinda…

its odd to me that it doesn't explicitly mention 'order of operations'.

Re: I quiz ChatGPT about math

#48

Post by CuriouslyC »

LWIRVoltage wrote: post #37

After reading so much here and this , about chatGPT- I have a feeling what will be needed for AI- is teachers. As it gets smarter and actually gets better reasoning, and capability- the way to keep AI sane and kind, will be to try to teach it like a person, with it putting importance on lessons from designated teachers even more, than it's weight in casual conversation , generally. I think i forsee teachers being hir…

You mean better data set curation and annotation? That's pretty much how you teach AI.

It would be funny to have a chatbot interface to the chatbot training dataset.

Re: I quiz ChatGPT about math

#49

Post by CrazyStat »

HDThoreaun wrote: post #19
CrazyStat wrote: post #3

I quizzed ChatGPT about math a few days back. It correctly answers that 1023 is odd, correctly answers that the square of an odd number is also odd, and is about 50/50 on whether it decides 1023^2 is even or odd. The author of the linked post repeatedly refers to ChatGPT's "reasoning". But ChatGPT doesn't reason, it just generates text. Sometimes that text might look like reasoning to a human--it's the sort of text t…

Your 5 year old is doing number theory? I think I was still working on like counting when I was 5

My five year old knows about evens and odds and squares, if that counts as number theory. He's really into multiplication.

Re: I quiz ChatGPT about math

#50

Post by dxbydt »

jfzoid wrote: post #39

I just tried doing this myself JFZ: how do we know there are infinitely many primes ChatGPT: One way to show that there are infinitely many prime numbers is to use the fact that the set of prime numbers is infinite to show that there must be an infinite number of composite numbers as well. Composite numbers are positive integers that have at least one positive integer divisor other than 1 and itself. For example, 4 i…

Here's a terribly amusing one on Z3. https://imgur.com/a/RZwHkLV

First it says I don't know the answer to 1+2. Then it says ofcourse 1+2 must be 1. In either case, its the same exact question, and if you assert closure on 3 elements, with 1+1=3 and 1+3=2, you are left with no choice other than 1+2=1 because closure and Cayley Hamilton. The very first property it lists is closure! So it knows what closure is, but doesn't know how to use it. But give it the answer to 2+1 and suddenly it uses associativity !

Post reply on HN