Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT-4o vs. Math

sabrina.dev

Post reply on HN
Print view View original

61–70 of 182 posts

Re: ChatGPT-4o vs. Math

#61

Post by [deleted] »

afro88 wrote: post #42
thomashop wrote: post #38

This recent article on Hacker News seems to suggest similar inconsistencies. GPT-4 Turbo with Vision is a step backward for coding (aider.chat) https://news.ycombinator.com/item?id=39985596 Without looking deeply at how cross-attention works, I imagine the instruction tuning of the multimodal models to be challenging. Maybe the magic is in synthetically creating this instruct dataset that combines images and text in…

The same guy found 4o to be much better GPT-4o takes #1 and #2 on the Aider LLM leaderboards https://news.ycombinator.com/item?id=40349655 Subjectively, I've found Aider to be much more useful on 4o. It still makes mistakes applying changes to files occasionally, but not so much to make me give up on it.

[deleted]

Re: ChatGPT-4o vs. Math

#62

Post by EmilyHughes »

cchance wrote: post #47

I'm fucking sorry but if you gave me that tape math problem i would have given the same answer! I'm so sick of people writing trick questions for AI's and then being like SEEEEEE it failed! And its like no you gave it data and a question and asked it to solve the question, it gave you the best answer it had... Like wtf. And i'm pretty sure the average person when asked would say the same thing and be like "duh" even…

How is this a trick question? Maybe I am dumb but I would have no idea how to solve this.

Re: ChatGPT-4o vs. Math

#63

Post by ben_w »

12907835202 wrote: post #35
CooCooCaCha wrote: post #33

Earlier quoted context omitted.

This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.

Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…

> Does anyone know how far off we are having logical AI?

1847, wasn't it? (George Boole). Or 1950-60 (LISP) or 1989 (Coq) depending on your taste?

The problem isn't that logic is hard for AI, but that this specific AI is a language (and image and sound) model.

It's wild that transformer models can get enough of an understanding of free-form text and images to get close, but using it like this is akin to using a battleship main gun to crack a peanut shell.

(Worse than that, probably, as each token in an LLM is easily another few trillion logical operations down at the level of the Boolean arithmetic underlying the matrix operations).

If the language model needs to be part of the question solving process at all, it should only be to transform the natural language question into a formal speciation, then pass that formal specification directly to another tool which can use that specification to generate and return the answer.

Re: ChatGPT-4o vs. Math

#64

Post by auggierose »

dunefox wrote: post #56
glial wrote: post #43

Earlier quoted context omitted.

I think LLMs will need to do what humans do: invent symbolic representations of systems and then "reason" by manipulating those systems according to rules. Here's a paper working along those lines: https://arxiv.org/abs/2402.03620

Is this what humans do?

That's what I am doing. I follow my intuition, but check it with logic.

Re: ChatGPT-4o vs. Math

#65

Post by slushy-chivalry »

ryanianian wrote: post #53
slushy-chivalry wrote: post #45

Earlier quoted context omitted.

maybe theorem proving could help? ask gpt4o to produce a proof in coq and see if it checks out...or split it into multiple agents -- one produces the proof of the closed formula for the tape roll thickness, and another one verifies it

Sure, but those are heuristics and feedback loops. They are not guaranteed to give you a solution. An LLM can never be a SAT solver unless it's an LLM with a SAT solver bolted on.

I don't disagree -- there is a place for specialized tool, and LLM wouldn't be my first pick if somebody asked me to add two large numbers.

There is nothing wrong with LLM + SAT solver -- especially if for an end-user it feels like they have 1 tool that solves their problem (even if under the hood it's 500 specialized tools governed by LLM).

My point about producing a proof was more about exploratory analysis -- sometimes reading (even incorrect) proofs can give you an idea for an interesting solution. Moreover, LLM can (potentially) spit out a bunch of possibly solutions and have another tool prune and verify and rank the most promising ones.

Also, the problem described in the blog is not a decision problem, so I'm not sure if it should be viewed through the lenses of computational complexity.

Re: ChatGPT-4o vs. Math

#66

Post by [deleted] »

cchance wrote: post #47

I'm fucking sorry but if you gave me that tape math problem i would have given the same answer! I'm so sick of people writing trick questions for AI's and then being like SEEEEEE it failed! And its like no you gave it data and a question and asked it to solve the question, it gave you the best answer it had... Like wtf. And i'm pretty sure the average person when asked would say the same thing and be like "duh" even…

[deleted]

Re: ChatGPT-4o vs. Math

#67

Post by giaour »

bombadilo wrote: post #7
Chinjut wrote: post #4

Earlier quoted context omitted.

It's a choke because it failed to get the answer. Saying other true things but not getting the answer is not a success.

I mean, in this context I agree. But most people doing math in high school or university are graded on their working of a problem, with the final result usually equating to a small proportion of the total marks received.

This depends on the grader and the context. Outside of an academic setting, sometimes being close to the right answer is better than nothing, and sometimes it is much worse. You can expect a human to understand which contexts require absolute precision and which do not, but that seems like a stretch for an LLM.

Re: ChatGPT-4o vs. Math

#68

Post by CooCooCaCha »

MR4D wrote: post #44
12907835202 wrote: post #35

Earlier quoted context omitted.

Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…

> Does anyone know how far off we are having logical AI? Your comment made me think of something. How do we know that logic AI is relevant? I mean, how do we know that humans are logic-AI driven and not statistical-intelligent?

A smart human can write and iterate on long, complex chains of logic. We can reason about code bases that are thousands of lines long.

Re: ChatGPT-4o vs. Math

#69

Post by krsrhe »

sabrina_ramonov wrote: post #29
lupire wrote: post #2

Need to run this experiment on a problem that is t already on its training set.

sometimes for physics/math exams, we'd get to create our own 1 pager cheat sheet to use. I'd just cram tons of actual problems/solutions on there, then scan for similarity.

[dead]

Re: ChatGPT-4o vs. Math

#70

Post by slushy-chivalry »

g9yuayon wrote: post #59

I actually have a contrarian view: being able to do elementary math is not that important in the current stage. Yes, understanding elementary math is a cornerstone for an AI to become more intelligent, but also let's be honest: LLMs are far from being AGIs and does not have common sense nor general ability to deduce or induct. If we accept such limitation of LLM, then focusing the mathematical understanding of an LLM…

if you sampled N random people on the street and asked them to solve this problem, what would the outcome be? would it be better than asking chatgpt N times? I wonder
Post reply on HN