Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can AI do maths yet? Thoughts from a mathematician

xenaproject.wordpress.com

Post reply on HN
Print view View original

71–80 of 364 posts

Re: Can AI do maths yet? Thoughts from a mathematician

#71

Post by spacemanspiff01 »

jampekka wrote: post #19

I just spent a few days trying to figure out some linear algebra with the help of ChatGPT. It's very useful for finding conceptual information from literature (which for a not-professional-mathematician at least can be really hard to find and decipher). But in the actual math it constantly makes very silly errors. E.g. indexing a vector beyond its dimension, trying to do matrix decomposition for scalars and insisting…

I wonder if these are tokenization issues? I really am curious about metas byte tokenization scheme...

Re: Can AI do maths yet? Thoughts from a mathematician

#72

Post by fn-mote »

4ad wrote: post #57

> FrontierMath is a secret dataset of “hundreds” of hard maths questions, curated by Epoch AI, and announced last month. The database stopped being secret when it was fed to proprietary LLMs running in the cloud. If anyone is not thinking that OpenAI has trained and tuned O3 on the "secret" problems people fed to GPT-4o, I have a bridge to sell you.

This level of conspiracy thinking requires evidence to be useful.

Edit: I do see from your profile that you are a real person though, so I say this with more respect.

Re: Can AI do maths yet? Thoughts from a mathematician

#73

Post by upghost »

jvanderbot wrote: post #69
upghost wrote: post #65

I didn't see anyone else ask this but.. isn't the FrontierMath dataset compromised now? At the very least OpenAI now knows the questions if not the answers. I would expect that the next iteration will "magically" get over 80% on the FrontierMath test. I imagine that experiment was pretty closely monitored.

I figured their model was independently evaluated against the questions/answers. That's not to say it's not compromised by "Here's a bag of money" type methods, but I don't even think it'd be a reasonable test if they just handed over the dataset.

I'm sure it was independently evaluated, but I'm sure the folks running the test were not given an on-prem installation of ChatGPT to mess with. It was still done via API calls, presumably through the chat interface UI.

That means the questions went over the fence to OpenAI.

I'm quite certain they are aware of that, and it would be pretty foolish not to take advantage of at least knowing what the questions are.

Re: Can AI do maths yet? Thoughts from a mathematician

#74

Post by sebzim4500 »

reshlo wrote: post #43
sebzim4500 wrote: post #42

Earlier quoted context omitted.

I can't reliably multiply four digit numbers in my head either, what's your point?

Nobody said you have to do it in your head.

That's the equivalent to what we are asking the model to do. If you give the model a calculator it will get 100%. If you give it a pen and paper (e.g. let it show it's working) then it will get near 100%.

Re: Can AI do maths yet? Thoughts from a mathematician

#75

Post by sincerecook »

No it can't, and there's no such thing as AI. How is a thing that predicts the next-most-likely word going to do novel math? It can't even do existing math reliably because logical operations and statistical approximation are fundamentally different. It is fun watching grifters put lipstick on this thing and shop it around as a magic pig though.

Re: Can AI do maths yet? Thoughts from a mathematician

#76

Post by retrocryptid »

When did we decide that AI == LLM? Oh don't answer. I know, The VC world noticed CNNs and LLMs about 10 years ago and it's the only thing anyone's talked about ever since.

Seems to me the answer to 'Can AI do maths yet?' depends on what you call AI and what you call maths. Our old departmental VAX running at a handfull of megahertz could do some very clever symbol manipulation on binomials and if you gave it a few seconds, it could even do something like theorum proving via proto-prolog. Neither are anywhere close to the glorious GAI future we hope to sell to industry and government, but it seems worth considering how they're different, why they worked, and whether there's room for some hybrid approach. Do LLMs need to know how to do math if they know how to write Prolog or Coc statements that can do interesting things?

I've heard people say they want to build software that emulates (simulates?) how humans do arithmetic, but ask a human to add anything bigger than two digit numbers and the first thing they do is reach for a calculator.

Re: Can AI do maths yet? Thoughts from a mathematician

#77

Post by jvanderbot »

upghost wrote: post #73
jvanderbot wrote: post #69

Earlier quoted context omitted.

I figured their model was independently evaluated against the questions/answers. That's not to say it's not compromised by "Here's a bag of money" type methods, but I don't even think it'd be a reasonable test if they just handed over the dataset.

I'm sure it was independently evaluated, but I'm sure the folks running the test were not given an on-prem installation of ChatGPT to mess with. It was still done via API calls, presumably through the chat interface UI. That means the questions went over the fence to OpenAI. I'm quite certain they are aware of that, and it would be pretty foolish not to take advantage of at least knowing what the questions are.

Now that you put it that way, it is laughably easy.

Re: Can AI do maths yet? Thoughts from a mathematician

#78

Post by jampekka »

spacemanspiff01 wrote: post #71
jampekka wrote: post #19

I just spent a few days trying to figure out some linear algebra with the help of ChatGPT. It's very useful for finding conceptual information from literature (which for a not-professional-mathematician at least can be really hard to find and decipher). But in the actual math it constantly makes very silly errors. E.g. indexing a vector beyond its dimension, trying to do matrix decomposition for scalars and insisting…

I wonder if these are tokenization issues? I really am curious about metas byte tokenization scheme...

Probably mostly not. The errors tend to be logical/conceptual. E.g. mixing up scalars and matrices is unlikely to be from tokenization. Especially if using spaces between the variables and operators, as AFAIK GPTs don't form tokens over spaces (although tokens may start or end with them).

Re: Can AI do maths yet? Thoughts from a mathematician

#80

Post by casenmgreen »

mdp2021 wrote: post #54
casenmgreen wrote: post #4

I may be wrong, but I think it a silly question. AI is basically auto-complete. It can do math to the extent you can find a solution via auto-complete based on an existing corpus of text.

> AI is basically Very many things conventionally labelled in the 50's. You are speaking of LLMs.

Yes - I mean only to say "AI" as the term is commonly used today.
Post reply on HN