Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT-4o vs. Math

sabrina.dev

Post reply on HN
Print view View original

91–100 of 182 posts

Re: ChatGPT-4o vs. Math

#91

Post by rthnbgrredf »

CooCooCaCha wrote: post #33
jcims wrote: post #21

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error. I then followed up with 'Your math is good but you derived incorrect data from the image. Can you take another look and see if you can tell where the error is?'. It figured it out and corrected it: Let's re-examine the image and the data provided: * The inner radius r1 is given as 5cm * The outer radius r2 is given as 1…

This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.

Statistically-based reasoning also applies to humans. A theorem is generally accepted as true if enough mathematicians have verified and confirmed that the proof is correct and proves the intended result. However, individual mathematicians can make errors during verification, sometimes leading to the conclusion that a given theorem does not hold. Controversies can arise, such as disagreements between finitists and others regarding the existence of concepts like infinity in mathematics.

Re: ChatGPT-4o vs. Math

#92

Post by hatenberg »

jcims wrote: post #21

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error. I then followed up with 'Your math is good but you derived incorrect data from the image. Can you take another look and see if you can tell where the error is?'. It figured it out and corrected it: Let's re-examine the image and the data provided: * The inner radius r1 is given as 5cm * The outer radius r2 is given as 1…

Chain of thought is nothing more than limiting the probability space enough that the model can provide the most likely answer. It's too much damn work to be useful.

Re: ChatGPT-4o vs. Math

#93

Post by mvdtnz »

It's incredible that we (humanity) are expending trillions of dollars and untold carbon emissions into these misinformation machines. I don't even mean machines for intentional generating misinformation (although they are that, too) but machines that we know misinform well-meaning users.

Peak humanity.

Re: ChatGPT-4o vs. Math

#94

Post by throwaway4aday »

12907835202 wrote: post #35
CooCooCaCha wrote: post #33

Earlier quoted context omitted.

This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.

Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…

Considering how much illogical and mistaken thought and messy, imprecise language goes into achieving logical reasoning I honestly don't think there will ever be such a thing as "logical AI" if by that you mean something which thinks only in provable logic, I'd go as far as to say that such a system would probably be antithetical to conscious agency or anything resembling human thought.

Re: ChatGPT-4o vs. Math

#95

Post by mvdtnz »

slushy-chivalry wrote: post #57
barkingcat wrote: post #52

Earlier quoted context omitted.

Have you tried actual math tools like wolfram alpha? My response to attempts to use llms to do math is usually, use the right tool for the job.

"a PhD level understanding of computer aided geometric design" could you elaborate? I found chatgpt to pretty useful in high level understanding of where the topics stand in the grand scheme of things kinda like a pre-google -- it helps me navigate what to google let's ask gpt4o You're an expert math researcher. - Give me an overview of topos theory - What are the main unsolved problems - What is the point of it? - W…

Please don't pollute comment sections with gpt output.

Re: ChatGPT-4o vs. Math

#96

Post by yatz »

jcims wrote: post #21

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error. I then followed up with 'Your math is good but you derived incorrect data from the image. Can you take another look and see if you can tell where the error is?'. It figured it out and corrected it: Let's re-examine the image and the data provided: * The inner radius r1 is given as 5cm * The outer radius r2 is given as 1…

Once you correct the LLM, it will continue to provide the corrected answer until some time later, when it will again make the same mistake. At least, this has been my experience. If you are using LLM to pull answers programmatically and rely on their accuracy, here is what worked for the structured or numeric answers, such as numbers, JSON, etc.

1) Send the same prompt twice, including "Can you double check?" in the second prompt to force GPT to verify the answer. 2) If both answers are the same, you got the correct answer. 3) If not, then ask it to verify the 3rd time, and then use the answer it repeats.

Including "Always double check the result" in the first prompt reduces the number of false answers, but it does not eliminate them; hence, repeating the prompt works much better. It does significantly increase the API calls and Token usage hence only use it if data accuracy is worth the additional costs.

Re: ChatGPT-4o vs. Math

#98

Post by OxfordOutlander »

slushy-chivalry wrote: post #82
d13 wrote: post #79

I have a theory that the more you use ChatGPT, the worse it becomes due to silent rate limiting - farming the work out to smaller quantized versions if you ask it a lot of questions. I’d like to see if the results of these tests are the same if you only ask one question per day.

that's an interesting hypothesis, I suppose one can make N calls to the API and look if the distribution of wrong answers is skewed towards the later portion of the API calls

I wouldnt expect this from the API, because each token is the same revenue for OAI. With chatGPT however, you pay a flat rate, so every incremental usage of it is a net-negative for them.

Re: ChatGPT-4o vs. Math

#99

Post by IanCal »

afro88 wrote: post #42
thomashop wrote: post #38

This recent article on Hacker News seems to suggest similar inconsistencies. GPT-4 Turbo with Vision is a step backward for coding (aider.chat) https://news.ycombinator.com/item?id=39985596 Without looking deeply at how cross-attention works, I imagine the instruction tuning of the multimodal models to be challenging. Maybe the magic is in synthetically creating this instruct dataset that combines images and text in…

The same guy found 4o to be much better GPT-4o takes #1 and #2 on the Aider LLM leaderboards https://news.ycombinator.com/item?id=40349655 Subjectively, I've found Aider to be much more useful on 4o. It still makes mistakes applying changes to files occasionally, but not so much to make me give up on it.

Anecdotally 4o has been working much better for coding for me, building things right the first time with less prodding. It may be a small shift in performance but it crosses a threshold where it's now useful enough and fast enough to be different from turbo.

Re: ChatGPT-4o vs. Math

#100

Post by jiiam »

slushy-chivalry wrote: post #70
g9yuayon wrote: post #59

I actually have a contrarian view: being able to do elementary math is not that important in the current stage. Yes, understanding elementary math is a cornerstone for an AI to become more intelligent, but also let's be honest: LLMs are far from being AGIs and does not have common sense nor general ability to deduce or induct. If we accept such limitation of LLM, then focusing the mathematical understanding of an LLM…

if you sampled N random people on the street and asked them to solve this problem, what would the outcome be? would it be better than asking chatgpt N times? I wonder

I am deeply interested in this point of view of yours so I will be hijacking your reply to ask another question: is "better than asking a few random people on the street" the bar we should be setting?

As far as mathematical thinking goes this doesn't seem an interesting metric at all. Do you believe that optimizing for this metric will indeed lead to reliable mathematical thinking?

I am of the idea that LLMs are not suited to maths, but since I'm not an expert of the field I'm always looking for counterarguments. Of course we can always wait another couple of years and the question will be resolved.

Post reply on HN