Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can AI do maths yet? Thoughts from a mathematician

xenaproject.wordpress.com

Post reply on HN
Print view View original

341–350 of 364 posts

Re: Can AI do maths yet? Thoughts from a mathematician

#341

Post by xigoi »

ls612 wrote: post #230
upghost wrote: post #73

Earlier quoted context omitted.

I'm sure it was independently evaluated, but I'm sure the folks running the test were not given an on-prem installation of ChatGPT to mess with. It was still done via API calls, presumably through the chat interface UI. That means the questions went over the fence to OpenAI. I'm quite certain they are aware of that, and it would be pretty foolish not to take advantage of at least knowing what the questions are.

Depending on the plan the researchers used they may have contractual protections against OpenAI training on their inputs.

Do you think ClosedAI cares about that? It’s not like anyone can tell if they violate the contract.

Re: Can AI do maths yet? Thoughts from a mathematician

#342

Post by skydhash »

caroline0v0 wrote: post #83

In fact, what I am most curious about is how AI understands symbolic logic relationships (neural networks and Turing machines are not completely equivalent). During training, this is a bunch of tokens.

I wouldn't say understand. But your answers is patterns. Formalism is mostly definition (axioms) and inference rules (theories). If we take programming languages, most grammars (which describe these two elements) are only a few pages long. With LLM being patterns seeker at its core, I guess it would be easy to extract the rules from a sample of programs, as the structure is so rigid.

You won't get the Turing machine evaluation mechanism and determinism, but you will have a generator. Although the viability of what is generated is is question. Because the other part of formalism, semantics, is almost always missing.

Re: Can AI do maths yet? Thoughts from a mathematician

#343

Post by skydhash »

alphan0n wrote: post #167
cruffle_duffle wrote: post #160

Earlier quoted context omitted.

It still has to know what to code in that environment. And based on my years of math as a wee little undergrad, the actual arithmetic was the least interesting part. LLM’s are horrible at basic arithmetic, but they can use python for the calculator. But python wont help them write the correct equations or even solve for the right thing (wolfram alpha can do a bit of that though)

You’ll have to show me what you mean. I’ve yet to encounter an equation that 4o couldn’t answer in 1-2 prompts unless it timed out. Even then it can provide the solution in a Jupyter notebook that can be run locally.

> I’ve yet to encounter an equation that 4o couldn’t answer in 1-2 prompts unless it timed out.

I've yet to encounter one of such sessions where the person is not handholding the LLMs during the whole process, basically describing the solution (instead of the problem) in natural language.

Re: Can AI do maths yet? Thoughts from a mathematician

#344

Post by afroboy »

ninetyninenine wrote: post #248
Syzygies wrote: post #247

"Can AI do math for us" is the canonical wrong question. People want self-driving cars so they can drink and watch TV. We should crave tools that enhance our abilities, as tools have done since prehistoric times. I'm a research mathematician. In the 1980's I'd ask everyone I knew a question, and flip through the hard bound library volumes of Mathematical Reviews, hoping to recognize something. If I was lucky, I'd get…

Your optimism should be tempered with the downside of progress meaning that AI in the near future may not only inspire creativity in humans, but it can replace human creativity all together. Why do I need to hire an artist for my movie/video game/advertisement when AI can replicate all the creativity I need.

Why do i need you as director if can ask AI to make movie for me?

Re: Can AI do maths yet? Thoughts from a mathematician

#345

Post by halayli »

vector_spaces wrote: post #316
halayli wrote: post #280

Earlier quoted context omitted.

these numbers are just your perception. The way you ask the question will very much influence the output and certain topics more than others. I get much better results when I share my certainty levels in my questions and say things like "if at all", "if any" etc.

I agree with this approach and use it myself, but these confidence markers can also skew output in undesirable ways. All of these heuristics are especially fragile when the subject matter touches the frontiers of what is known. In any case my best experiences with LLMs for pure math research have been for exploring the problem space and ideation -- queries along the line of "Here's a problem I'm working on ... . Do a…

That's exactly how I use it. I find claude way more plesent to use than any other gpt I've used.

Re: Can AI do maths yet? Thoughts from a mathematician

#346

Post by jampekka »

cheald wrote: post #323
jampekka wrote: post #19

I just spent a few days trying to figure out some linear algebra with the help of ChatGPT. It's very useful for finding conceptual information from literature (which for a not-professional-mathematician at least can be really hard to find and decipher). But in the actual math it constantly makes very silly errors. E.g. indexing a vector beyond its dimension, trying to do matrix decomposition for scalars and insisting…

LLMs have been very useful for me in explorations of linear algebra, because I can have an idea and say "what's this operation called?" or "how do I go from this thing to that thing?", and it'll give me the mechanism and an explanation, and then I can go read actual human-written literature or documentation on the subject. It often gets the actual math wrong, but it is good enough at connecting the dots between my la…

Very close to my experience.

Re: Can AI do maths yet? Thoughts from a mathematician

#347

Post by ninetyninenine »

afroboy wrote: post #344
ninetyninenine wrote: post #248

Earlier quoted context omitted.

Your optimism should be tempered with the downside of progress meaning that AI in the near future may not only inspire creativity in humans, but it can replace human creativity all together. Why do I need to hire an artist for my movie/video game/advertisement when AI can replicate all the creativity I need.

Why do i need you as director if can ask AI to make movie for me?

you dont. that's my point. Probably don't even need the person who hires the director.

Re: Can AI do maths yet? Thoughts from a mathematician

#348

Post by vintermann »

amanda99 wrote: post #266
Syzygies wrote: post #247

"Can AI do math for us" is the canonical wrong question. People want self-driving cars so they can drink and watch TV. We should crave tools that enhance our abilities, as tools have done since prehistoric times. I'm a research mathematician. In the 1980's I'd ask everyone I knew a question, and flip through the hard bound library volumes of Mathematical Reviews, hoping to recognize something. If I was lucky, I'd get…

> AI is often wrong, never knows when it's wrong, but people are like this too. When talking with various models of ChatGPT about research math, my biggest gripe is that it's either confidently right (10% of my work) or confidently wrong (90%). A human researcher would be right 15% of the time, unsure 50% of the time, and give helpful ideas that are right/helpful (25%) or wrong/a red herring (10%). And only 5% of the…

I think that is a lot about how it's tuned. It's optimized for questions which can be answered with one big answer with bullet points. It's also optimized for relatively easy questions that have clear and correct answers. I have yet to encounter a QA bot which will stop and ask clarifying questions before producing its big bullet point post of answers.

I think this is a sensible tuning in that it's probably what most people who log on to chatgpt want. Most questions people ask of it will have simple enough answers that require knowledge but not all that much reasoning.

But I see no reason why it couldn't be tuned to be more open ended, less eager to give the correct benchmark/exam answer right away. Indeed in the "internal narrative" of recent models, I see them ask themselves things I wish they asked me!

Re: Can AI do maths yet? Thoughts from a mathematician

#349

Post by a_petrov »

I use ChatGPT to help study linear algebra.

It helps me a lot when I feel lost. It's often wrong in the calculations, but it's cool to have a study buddy that doesn't judge you.

If I get blocked with a problem I can't solve, I ask for assistance with my approach.

I enjoy asking ChatGPT about the context behind all that math theory. It's nice to elaborate on that as most of the math books are very lean and provide no applied context.

Post reply on HN