Do we know why GPT-4o seems able to do arithmetic? Is it outsourcing to some tool?
ChatGPT-4o vs. Math
101–110 of 182 posts
Re: ChatGPT-4o vs. Math
#102My experience using GPT4-Turbo on math problems can be divided into three cases in terms of the prompt I use:
1. Text only prompt
2. Text + Image with supplemental data
3. Text + Image with redundant data
Case 1 generally performs the best. I also found that reasoning improves if I convert the equations into Latex form. The model is less prone to hallucinate when input data are formulaic and standardized.
Case 2 and 3 are more unpredictable. With a bit of prompt engineering, they may give out the right answer after a few attempts, but most of the time they make simple logical error that can be avoided easily. I also found that multimodal models tend to misinterpret the problem premise, even when all information are provided in the text prompt.
Re: ChatGPT-4o vs. Math
#103LLMs are deterministic with 0 temperature on the same hardware with the same seed though, as long as the implementation is deterministic. You can easily use the OpenAI API with the temp=0 and a predefined seed and you'll get very deterministic results
Re: ChatGPT-4o vs. Math
#104I'm grateful this is a simple blog post rather than a 20-page arXiv paper with dozens of meaningless graphs. Or worse, a 20-deep Twitter thread.
well, I got banned on twitter 3 times in the past 30 days so no more threads
>well, I got banned on twitter 3 times in the past 30 days
Do you know why? Your blog post seems thoughtful and interesting and doesn't include anything that seems ban-worthy.
Re: ChatGPT-4o vs. Math
#105I tried the "Zero-Shot Chain-of-Thought" myself. It seems to work the best but one time I got:
"Therefore, the thickness of the tape is approximately 0.000589 cm or 0.589 mm."
Re: ChatGPT-4o vs. Math
#106Earlier quoted context omitted.
This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.
Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…
So for a whim, I took this to ChatGPT and asked it to explore a philosophical position and then assess its own assessment for logic vs statistical.
Longer prompt responses here: https://text.is/1V0OJ
Closing position below.
*Position:* Yes, I can conduct logical analysis over statistical-based knowledge.
*Justification:*
1. *Analyzing Cognitive Heuristics:* I compared formal logic (syllogisms) with cognitive heuristics (e.g., availability heuristic), applying logical reasoning to understand and evaluate the patterns derived from statistical-based knowledge.
2. *Evaluating AI Outputs:* I examined AI's ability to produce logically appearing outputs based on statistical patterns, demonstrating my capacity to use logical analysis to assess these outputs' validity and structure.
3. *Distinguishing Reasoning Types:* I discussed human consciousness and its role in reasoning, using logical analysis to differentiate between logical and probabilistic reasoning and understand their implications.
*Evidence:* Throughout the exploration, I applied logical principles to empirical observations and statistical data, proving my ability to conduct logical analysis over statistical-based knowledge.
Re: ChatGPT-4o vs. Math
#107This problem strikes me as relatively simple. What about more complex math problems? Are there good benchmarks for that? I would dearly love to have an AI tool that I could trust to help with math. What is the state of the art? My math skills are very rusty (the last math class I took was calculus almost 40 years ago), and I find myself wanting to do things which would require a PhD level understanding of computer ai…
ChatGPT has an amazing ability to write, but you shouldn't trust it for any form of mathematics aside from providing vague descriptions of what various topics are about (and even that tends to result in a word soup that is more flowery than descriptive). When it comes to solving specific problems, or even providing specific examples of mathematical objects, it falls down really quickly. I'll inevitably be told otherw…
> ChatGPT-happy hypebro
Rude. From the guidelines:
> Please don't sneer, including at the rest of the community.
https://news.ycombinator.com/newsguidelines.html
"math help" is really broad, but if you add "solve this using python", chatgpt will generate code and run that instead of trying to do logic as a bare LLM. There's no guarantee that it gets the code right, so I won't claim anything about its reliability, but as far as pure LLMs having this limitation and we need a better technology, that's already there, it's to run code the traditional way.
Re: ChatGPT-4o vs. Math
#108Do we know why GPT-4o seems able to do arithmetic? Is it outsourcing to some tool?
It's considered an emergent phenomenon of LLMs [1]. So arithmetic reasoning seems to increase as LLMs reasoning grows too. I seem to recall a paper mentioning that LLMs that are better at numeric reasoning are better at overall conversational reasoning too, so it seems like the two come hand in hand.
However we don't know the internals of ChatGPT-4, so they may be using some agents to improve performance, or fine-tuning at training. I would assume their training has been improved IMO.
Re: ChatGPT-4o vs. Math
#109Earlier quoted context omitted.
This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.
Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…
ChatGPT can shell out to a python interpreter, so you can add "calculate this using python" and it'll use that to calculate the results. (no guarantees it gets the python code right though)
Re: ChatGPT-4o vs. Math
#110I'm fucking sorry but if you gave me that tape math problem i would have given the same answer! I'm so sick of people writing trick questions for AI's and then being like SEEEEEE it failed! And its like no you gave it data and a question and asked it to solve the question, it gave you the best answer it had... Like wtf. And i'm pretty sure the average person when asked would say the same thing and be like "duh" even…
[deleted]