Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT-4o vs. Math

sabrina.dev

Post reply on HN
Print view View original

111–120 of 182 posts

Re: ChatGPT-4o vs. Math

#111

Post by xienze »

usaar333 wrote: post #9
soarerz wrote: post #3

The model's first attempt is impressive (not sure why it's labeled a choke). Unfortunately gpt4o cannot discover calculus on its own.

Or.. use calculus? It has gotten quite impressive at handling calculus word problems. GPT-4 (original) failed miserably on this problem (attempted to set it up using constant acceleration equations); GPT-4O finally gets it correct: > I am driving a car at 65 miles per hour and release the gas pedal. The only force my car is now experiencing is air resistance, which in this problem can be assumed to be linearly propor…

Does it get the answer right every single time you ask the question the same way? If not, who cares how it’s coming to an answer, it’s not consistently correct and therefore not dependable. That’s what the article was exploring.

Re: ChatGPT-4o vs. Math

#112

Post by guitarlimeo »

I fed the chain-of-thought prompt to GTP-4o and got a correct answer back. I then got the idea to say that the answer was incorrect to see if it would recalculate and come back with the same answer. As you could guess already, it arrived on a completely different answer showing no ability of real logical reasoning.

Re: ChatGPT-4o vs. Math

#113

Post by MR4D »

CooCooCaCha wrote: post #68
MR4D wrote: post #44

Earlier quoted context omitted.

> Does anyone know how far off we are having logical AI? Your comment made me think of something. How do we know that logic AI is relevant? I mean, how do we know that humans are logic-AI driven and not statistical-intelligent?

A smart human can write and iterate on long, complex chains of logic. We can reason about code bases that are thousands of lines long.

But is that really logic?

For instance, we supposedly reason about complex driving laws, but for anyone who has run a stop light late at night when there is no other traffic is acting statistically, not logically.

Re: ChatGPT-4o vs. Math

#114

Post by nurple »

xanderlewis wrote: post #76
s1mon wrote: post #51

This problem strikes me as relatively simple. What about more complex math problems? Are there good benchmarks for that? I would dearly love to have an AI tool that I could trust to help with math. What is the state of the art? My math skills are very rusty (the last math class I took was calculus almost 40 years ago), and I find myself wanting to do things which would require a PhD level understanding of computer ai…

ChatGPT has an amazing ability to write, but you shouldn't trust it for any form of mathematics aside from providing vague descriptions of what various topics are about (and even that tends to result in a word soup that is more flowery than descriptive). When it comes to solving specific problems, or even providing specific examples of mathematical objects, it falls down really quickly. I'll inevitably be told otherw…

I'm with you. The thing I find baffling is how anyone with any logical sense finds chatGPT useful for anything that requires precision, like math and code. If you do indeed follow the caveats that the LLM companies require placing alongside any output: to not rely on it, and verify it yourself, then you already have to be skilled enough to detect problems, and if you are that skilled, the only way to check the output is to do the work again yourself!

So, umm, where's the savings? You can't not do the work to check the output, and a novice just can't check at all...

I have personally been brought into a coding project created by a novice using GPT4, and I was completely blown away by how bad the code was. I was asked to review the code because the novice dev just couldn't get the required functionality to work fully. Turns out that since he didn't understand the deployment platform, or networking, or indeed the language he was using, that there was actually no possible way to accomplish the task with the approach him and the LLM had "decided" on.

He had been working on that problem for three weeks. I leveraged 2 off-the-shelf tools and had a solve from scratch in under a full day's work, including integration testing.

Re: ChatGPT-4o vs. Math

#115

Post by logicallee »

As a human I couldn't solve it. I missed the key insight that we can calculate the side surface area and it will be the same if it is rolled out into a rectangle.

It might make more sense to give it math problems with enough hints that a human can definitely do it. For example you might try saying: "Here is an enormous hint: the side surface area is easy to calculate when it is rolled up and doesn't change when it is unrolled into a rectangle, so if you calculate the side surface area when rolled up you can then divide by the known length to get the width."

I think with such a hint I might have gotten it, and ChatGPT might have as well.

Another interesting thing is that when discussing rolls of tape we don't really talk about inner diameters that much so it doesn't have that much training data. Perhaps a simpler problem could have been something like "Imagine a roll of tape where the tape itself has constant thickness x and length y. The width of the tape doesn't matter for this problem. We will calculate the thickness. The roll of tape is completely rolled up into a perfectly solid circular shape and a diameter of z. What is the formula for the thickness of the tape x expressed in terms of length y and 'diameter of the tape when rolled up in a circle' z? In coming up with the formula use the fact that the constant thickness doesn't change when it is unrolled from a circular to a rectangular shape."

With so much handholding, (and using the two-dimensional word circular rather than calling it a cylinder and rectangular prism which is what it really is) many more people could apply the formula correctly and get the result. But can ChatGPT?

I just tested it, this is how it did:

https://chat.openai.com/share/ddd0eef3-f42f-4559-8948-e028da...

I can't follow its math so I don't know if it's right or not but it definitely didn't go straight for the simplified formula. (pi times half the diameter squared to get the area of the solid "circle" and divide by the length to get the thickness of the tape.)

Re: ChatGPT-4o vs. Math

#116

Post by flyingspaceship »

ianbicking wrote: post #27

Similar to the article, I haven't found complementary image data to be that useful. If the information is really missing without the image, then the image is useful. But if the basic information is all available textually (including things like the code that produces a diagram) then the image doesn't seem to add much except perhaps some chaos/unpredictability. But reading this I do have a thought: chain of thought, o…

The images bring with it their own unique set of problems. I was using it to help analyze UIs (before and after images) to determine if the changes I made were better or worse, but after using it for awhile I realized that it favored the second image in the comparison to an extent that made it difficult to tell which it thought was better. I suppose it's being trained on before and afters and generally the afters are always better!

Re: ChatGPT-4o vs. Math

#117

Post by phatfish »

giaour wrote: post #67
bombadilo wrote: post #7

Earlier quoted context omitted.

I mean, in this context I agree. But most people doing math in high school or university are graded on their working of a problem, with the final result usually equating to a small proportion of the total marks received.

This depends on the grader and the context. Outside of an academic setting, sometimes being close to the right answer is better than nothing, and sometimes it is much worse. You can expect a human to understand which contexts require absolute precision and which do not, but that seems like a stretch for an LLM.

LLMs being confidently incorrect until they are challenged is a bad trait. At least they have a system prompt to tell them to be polite about it.

Most people learn to avoid that person that is wrong/has bad judgment and is arrogant about it.

Re: ChatGPT-4o vs. Math

#118

Post by deely3 »

Tiberium wrote: post #103

LLMs are deterministic with 0 temperature on the same hardware with the same seed though, as long as the implementation is deterministic. You can easily use the OpenAI API with the temp=0 and a predefined seed and you'll get very deterministic results

> You can easily use the OpenAI API with the temp=0 and a predefined seed and you'll get very deterministic results

Does that mean that in this situation OpenAI will always answer wrongly for the same question?

Re: ChatGPT-4o vs. Math

#119

Post by MagicMoonlight »

g9yuayon wrote: post #59

I actually have a contrarian view: being able to do elementary math is not that important in the current stage. Yes, understanding elementary math is a cornerstone for an AI to become more intelligent, but also let's be honest: LLMs are far from being AGIs and does not have common sense nor general ability to deduce or induct. If we accept such limitation of LLM, then focusing the mathematical understanding of an LLM…

It’s important because solving a math problem requires you to actually understand something and follow deliberate steps.

The fact that they can’t means they’re just a toy ultimately.

Re: ChatGPT-4o vs. Math

#120

Post by 1970-01-01 »

>GPT-4o interprets “how thick is the tape” as referring to the cross-section of the tape roll, rather than the thickness of a piece of tape.

As someone that has tapes of varied "thickness", I was also confused for several minutes. I would give GPT partial credit on this attempt. Also note the author has implied (is biased toward finding) a piece of tape thickness and not the thickness of the entire object/roll.

https://m.media-amazon.com/images/I/71q3WQNl3nL._SL1500_.jpg

Post reply on HN