Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT-4o vs. Math

sabrina.dev

Post reply on HN
Print view View original

131–140 of 182 posts

Re: ChatGPT-4o vs. Math

#131

Post by xanderlewis »

nurple wrote: post #114
xanderlewis wrote: post #76

Earlier quoted context omitted.

ChatGPT has an amazing ability to write, but you shouldn't trust it for any form of mathematics aside from providing vague descriptions of what various topics are about (and even that tends to result in a word soup that is more flowery than descriptive). When it comes to solving specific problems, or even providing specific examples of mathematical objects, it falls down really quickly. I'll inevitably be told otherw…

I'm with you. The thing I find baffling is how anyone with any logical sense finds chatGPT useful for anything that requires precision, like math and code. If you do indeed follow the caveats that the LLM companies require placing alongside any output: to not rely on it, and verify it yourself, then you already have to be skilled enough to detect problems, and if you are that skilled, the only way to check the output…

> So, umm, where's the savings? You can't not do the work to check the output, and a novice just can't check at all...

You’re exactly right. It’s a weird example of a technology that is ridiculously impressive (at least at first impression, but also legitimately quite astounding) whilst also being seemingly useless.

I guess the oft-drawn parallels between AI and nuclear weapons are not (yet) that they’re both likely to lead to the apocalypse but more that they both represent era-defining achievements in science/technology whilst simultaneously being utterly unusable for anything productive.

At least nukes have the effect of deterring us from WW3…

Re: ChatGPT-4o vs. Math

#132

Post by Auracle »

jcims wrote: post #21

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error. I then followed up with 'Your math is good but you derived incorrect data from the image. Can you take another look and see if you can tell where the error is?'. It figured it out and corrected it: Let's re-examine the image and the data provided: * The inner radius r1 is given as 5cm * The outer radius r2 is given as 1…

That’s funny. I practically got into a shouting match for the first time ever with ChatGPT earlier today because I was asking it to create a function to make a filled circle of pixels of a certain size using diameter and absolutely not radius (with some other constraints).

This mattered because I wanted clear steps between 3,4,5,6 etc pixels wide, so the diameter was an int.

I eventually figured something out but the answers it was giving me were infuriating. At some point instead of a radius it put “int halfSize = diameter / 2”.

Re: ChatGPT-4o vs. Math

#133

Post by jiggawatts »

d0100 wrote: post #89
12907835202 wrote: post #35

Earlier quoted context omitted.

Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…

We could get there if current LLM's managed to prepare some data and offload it to a plugin, then continue on with the result * LLM extracts the problem and measurements * Sends the data to a math plugin * Continues its reasoning with the result

That’s already a thing. ChatGPT can utilise Wolfram Mathematica as a “tool”. Conversely, there’s an LLM included in the latest Mathematica release.

Re: ChatGPT-4o vs. Math

#134

Post by groby_b »

yatz wrote: post #96
jcims wrote: post #21

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error. I then followed up with 'Your math is good but you derived incorrect data from the image. Can you take another look and see if you can tell where the error is?'. It figured it out and corrected it: Let's re-examine the image and the data provided: * The inner radius r1 is given as 5cm * The outer radius r2 is given as 1…

Once you correct the LLM, it will continue to provide the corrected answer until some time later, when it will again make the same mistake. At least, this has been my experience. If you are using LLM to pull answers programmatically and rely on their accuracy, here is what worked for the structured or numeric answers, such as numbers, JSON, etc. 1) Send the same prompt twice, including "Can you double check?" in the…

> Once you correct the LLM, it will continue to provide the corrected answer until some time later,

That is only true if you stay within the same chat. It is not true across chats. Context caching is something that a lot of folks would really really like to see.

And jumping to a new chat is one of the core points of the OP: "I restarted with a slightly modified prompt:"

The iterations before where mostly to figure out why the initial prompt went wrong. And AFAICT there's a good insight in the modified prompt - "Make no assumptions". Probably also "ensure you fully understand how it's labelled".

And no, asking repeatedly doesn't necessarily give different answers, not even with "can you double check". There are quite a few examples where LLMs are consistently and proudly wrong. Don't use LLMs if 100% accuracy matters.

Re: ChatGPT-4o vs. Math

#135

Post by Melatonic »

ben_w wrote: post #63
12907835202 wrote: post #35

Earlier quoted context omitted.

Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…

> Does anyone know how far off we are having logical AI? 1847, wasn't it? (George Boole). Or 1950-60 (LISP) or 1989 (Coq) depending on your taste? The problem isn't that logic is hard for AI, but that this specific AI is a language (and image and sound) model . It's wild that transformer models can get enough of an understanding of free-form text and images to get close, but using it like this is akin to using a batt…

By that same logic isn't that a similar process that we humans use as well ? Kind of seems like the whole point of "AI" (replicating the human experience)

Re: ChatGPT-4o vs. Math

#136

Post by mmmmmmmike »

tapeaway wrote: post #128

Isn't there an unstated simplification here that: * the tape is perfectly flexible * the tape has been rolled with absolutely no gap between layers?

Yeah, and even given that, there’s the question of how exactly it deforms from its flattened shape to make a spiral (and if this changes the area). I wouldn’t agree with the “correct” answer if the tape was very thick, but given that the answer is .005 cm, it’s probably thin enough that such an approximation is okay.

Re: ChatGPT-4o vs. Math

#137

Post by jiggawatts »

ilaksh wrote: post #127

If you really want to see what the SOTA model can do, look at the posts on the web page for the mind-blowing image output. That is not released yet. https://openai.com/index/hello-gpt-4o/ Mark my words, that is the sort of thing that Ilya saw months ago and I believe he decided they had achieved their mission of AGI. And so that would mean stopping work, giving it to the government to study, or giving it away or some…

I like your theory but if it’s true, then Ilya was wrong.

All of the current LLM architectures have no medium-term memory or iterative capability. That means they’re missing essential functionality for general intelligence.

I tired GPT 4o for various tasks and it’s good but it isn’t blowing my skirt up. The only noticeable difference is the speed, which is a very nice improvement that enables new workflows.

Re: ChatGPT-4o vs. Math

#138

Post by jiggawatts »

jiiam wrote: post #100
slushy-chivalry wrote: post #70

Earlier quoted context omitted.

if you sampled N random people on the street and asked them to solve this problem, what would the outcome be? would it be better than asking chatgpt N times? I wonder

I am deeply interested in this point of view of yours so I will be hijacking your reply to ask another question: is "better than asking a few random people on the street" the bar we should be setting? As far as mathematical thinking goes this doesn't seem an interesting metric at all. Do you believe that optimizing for this metric will indeed lead to reliable mathematical thinking? I am of the idea that LLMs are not…

People compare a general intelligence against the yardstick of their own specialist skills.

I’ve seen some truly absurd examples, like people complaining that it didn’t have the latest updates to some obscure research functional logic proof language that has maybe a hundred users globally!

GPT 4 already has markedly superior English comprehension and basic logic than most people I interact with on a daily basis. It’s only outperformed by a handful of people, all of whom are “high achievers” such as entrepreneurs, professors, or consultants.

I actively simplify my speech when talking to ordinary people to avoid overwhelming them. I don’t need to when instructing GPT.

Re: ChatGPT-4o vs. Math

#139

Post by ilaksh »

jiggawatts wrote: post #137
ilaksh wrote: post #127

If you really want to see what the SOTA model can do, look at the posts on the web page for the mind-blowing image output. That is not released yet. https://openai.com/index/hello-gpt-4o/ Mark my words, that is the sort of thing that Ilya saw months ago and I believe he decided they had achieved their mission of AGI. And so that would mean stopping work, giving it to the government to study, or giving it away or some…

I like your theory but if it’s true, then Ilya was wrong. All of the current LLM architectures have no medium-term memory or iterative capability. That means they’re missing essential functionality for general intelligence. I tired GPT 4o for various tasks and it’s good but it isn’t blowing my skirt up. The only noticeable difference is the speed, which is a very nice improvement that enables new workflows.

Part of the confusion is that people use the term "AGI" to mean different things. We should actually call this AGI, because it is starkly different from the narrow capabilities of AI a few years ago.

I am not claiming that it is a full digital simulation of a human being or has all of the capabilities of animals like humans, or is the end of intelligence research. But it is obviously very general purpose at this point, and very human-like in many ways.

Study this page carefully: https://openai.com/index/hello-gpt-4o/ .. much of that was deliberately omitted from the presentation.

Re: ChatGPT-4o vs. Math

#140

Post by jiggawatts »

s1mon wrote: post #80
barkingcat wrote: post #52

Earlier quoted context omitted.

Have you tried actual math tools like wolfram alpha? My response to attempts to use llms to do math is usually, use the right tool for the job.

I have tried to use Wolfram Alpha inside of ChatGPT, but that didn't get me very far. It seems like I would need to understand a lot more math to be able to do anything useful with Wolfram Alpha, and perhaps it would be better to run it stand alone not as a plugin.

Ask it to write you the Wolfram language code and then verify it and execute it yourself.

I’ve found that I can work 100x faster with Mathematica this way and solve problems that I wouldn’t have bothered to attempt otherwise.

This is particularly effective for quickly visualising things, I’m too lazy to figure out all the graphing options for esoteric scenarios but GPT 4 can quickly iterate over variants given feedback.

Post reply on HN