Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT-4o vs. Math

sabrina.dev

Post reply on HN
Print view View original

71–80 of 182 posts

Re: ChatGPT-4o vs. Math

#71

Post by xanderlewis »

12907835202 wrote: post #35
CooCooCaCha wrote: post #33

Earlier quoted context omitted.

This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.

Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…

> Math seems like low hanging fruit in that regard.

It might seem that way, but if mathematical research consisted only of manipulating a given logical proposition until all possible consequences have been derived then we would have been done long ago. And we wouldn't need AI (in the modern sense) to do it.

Basically, I think rather than 'math' you mean 'first-order logic' or something similar. The former is a very, large superset of the latter.

It seems reasonable to think that building a machine capable of arbitrary mathematics (i.e. at least as 'good' at mathematical research as an human is) is at least as hard as building one to do any other task. That is, it might as well be the definition of AGI.

Re: ChatGPT-4o vs. Math

#73

Post by Chinjut »

slushy-chivalry wrote: post #57
barkingcat wrote: post #52

Earlier quoted context omitted.

Have you tried actual math tools like wolfram alpha? My response to attempts to use llms to do math is usually, use the right tool for the job.

"a PhD level understanding of computer aided geometric design" could you elaborate? I found chatgpt to pretty useful in high level understanding of where the topics stand in the grand scheme of things kinda like a pre-google -- it helps me navigate what to google let's ask gpt4o You're an expert math researcher. - Give me an overview of topos theory - What are the main unsolved problems - What is the point of it? - W…

That's fine but it's about the same as you'll get from an encyclopedia also, which makes sense as that's just where GPT got it from anyway. Nothing revolutionary in the ability to read encyclopedia articles. We've had that forever.

Re: ChatGPT-4o vs. Math

#74

Post by slushy-chivalry »

EmilyHughes wrote: post #62
cchance wrote: post #47

I'm fucking sorry but if you gave me that tape math problem i would have given the same answer! I'm so sick of people writing trick questions for AI's and then being like SEEEEEE it failed! And its like no you gave it data and a question and asked it to solve the question, it gave you the best answer it had... Like wtf. And i'm pretty sure the average person when asked would say the same thing and be like "duh" even…

How is this a trick question? Maybe I am dumb but I would have no idea how to solve this.

to be fair, this question does not require any advanced math beyond knowing how to compute the area of a disk

to me, the impressive part of gpt is being able to understand the image and extract data from it (radius information) and come up with an actual solution (even though it got it wrong a few times)

for basic math I can do

  python -c "print(6/9)"

Re: ChatGPT-4o vs. Math

#75

Post by jamilton »

slushy-chivalry wrote: post #45
ryanianian wrote: post #36

Earlier quoted context omitted.

(Not an AI researcher, just someone who likes complexity analysis.) Discrete reasoning is NP-Complete. You can get very close with the stats-based approaches of LLMs and whatnot, but your minima/maxima may always turn out to be local rather than global.

maybe theorem proving could help? ask gpt4o to produce a proof in coq and see if it checks out...or split it into multiple agents -- one produces the proof of the closed formula for the tape roll thickness, and another one verifies it

I had the thought recently that theorem provers could be a neat source of synthetic data. Make an LLM generate a proof, run it to evaluate it and label it as valid/invalid, fine-tune the LLM on the results. In theory it should then more consistently create valid proofs.

Re: ChatGPT-4o vs. Math

#76

Post by xanderlewis »

s1mon wrote: post #51

This problem strikes me as relatively simple. What about more complex math problems? Are there good benchmarks for that? I would dearly love to have an AI tool that I could trust to help with math. What is the state of the art? My math skills are very rusty (the last math class I took was calculus almost 40 years ago), and I find myself wanting to do things which would require a PhD level understanding of computer ai…

ChatGPT has an amazing ability to write, but you shouldn't trust it for any form of mathematics aside from providing vague descriptions of what various topics are about (and even that tends to result in a word soup that is more flowery than descriptive). When it comes to solving specific problems, or even providing specific examples of mathematical objects, it falls down really quickly.

I'll inevitably be told otherwise by some ChatGPT-happy hypebro, but LLMs are hopeless when it comes to anything requiring reasoning. Scaling it up will lessen the chance of a cock-up, but anything vaguely out of distribution will result in the same nonsense we're all used to by now. Those who say otherwise very likely just lack the experience or knowledge necessary to challenge the model enough or interpret the results.

As a test of this claim: please comment below if you, say, have a degree in mathematics and believe LLMs to be reliable for 'math help' (and explain why you think so).

We need a better technology! And when this better technology finally comes along, we'll look back at pure LLMs and laugh about how we ever believed we could magic such a machine into existence just by pouring data into a model originally designed for machine translation.

Re: ChatGPT-4o vs. Math

#77

Post by slushy-chivalry »

Chinjut wrote: post #73
slushy-chivalry wrote: post #57

Earlier quoted context omitted.

"a PhD level understanding of computer aided geometric design" could you elaborate? I found chatgpt to pretty useful in high level understanding of where the topics stand in the grand scheme of things kinda like a pre-google -- it helps me navigate what to google let's ask gpt4o You're an expert math researcher. - Give me an overview of topos theory - What are the main unsolved problems - What is the point of it? - W…

That's fine but it's about the same as you'll get from an encyclopedia also, which makes sense as that's just where GPT got it from anyway. Nothing revolutionary in the ability to read encyclopedia articles. We've had that forever.

sure, but with like a 100x improvement in usability -- chatgpt is helpful in figuring out what stuff to read (at least for me) so that when I go to the actual paper or a book I know what to focus on

otherwise you can say "why do you need google, it's the same as you'll get from the website"

moreover, I found that chatgpt is pretty decent at rephrasing a convoluted concept or a paragraph in a research paper, or even giving me ideas on the research directions

I mean, same with coding -- I treat it as a smart autocomplete

I could go to google and look for a .csv containing a list of all US States

Or, I can write

  const US_STATES = [
and let copilot complete it for me -- 5 minutes saved?

Re: ChatGPT-4o vs. Math

#78

Post by entropicdrifter »

ben_w wrote: post #63
12907835202 wrote: post #35

Earlier quoted context omitted.

Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actuall…

> Does anyone know how far off we are having logical AI? 1847, wasn't it? (George Boole). Or 1950-60 (LISP) or 1989 (Coq) depending on your taste? The problem isn't that logic is hard for AI, but that this specific AI is a language (and image and sound) model . It's wild that transformer models can get enough of an understanding of free-form text and images to get close, but using it like this is akin to using a batt…

Right? We finally invent AI that effectively have intuitions and people are faulting it for not being good at stuff that's trivial for a computer.

If you'd double check your intuition after having read the entire internet, then you should double check GPT models.

Re: ChatGPT-4o vs. Math

#79

Post by d13 »

I have a theory that the more you use ChatGPT, the worse it becomes due to silent rate limiting - farming the work out to smaller quantized versions if you ask it a lot of questions. I’d like to see if the results of these tests are the same if you only ask one question per day.

Re: ChatGPT-4o vs. Math

#80

Post by s1mon »

barkingcat wrote: post #52
s1mon wrote: post #51

This problem strikes me as relatively simple. What about more complex math problems? Are there good benchmarks for that? I would dearly love to have an AI tool that I could trust to help with math. What is the state of the art? My math skills are very rusty (the last math class I took was calculus almost 40 years ago), and I find myself wanting to do things which would require a PhD level understanding of computer ai…

Have you tried actual math tools like wolfram alpha? My response to attempts to use llms to do math is usually, use the right tool for the job.

I have tried to use Wolfram Alpha inside of ChatGPT, but that didn't get me very far. It seems like I would need to understand a lot more math to be able to do anything useful with Wolfram Alpha, and perhaps it would be better to run it stand alone not as a plugin.
Post reply on HN