Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.6 used a prompt to close a 30-year gap in convex optimization

old.reddit.com

Post reply on HN
Print view View original

361–370 of 414 posts

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#361

Post by BobbyTables2 »

xeromal wrote: post #165
_alternator_ wrote: post #76

I know a bit about this field. This conjecture reads as somewhat more niche than the cyclic double cover conjecture recently proved by OpenAI, but nevertheless represents a real contribution. You want to know how long it takes to solve an optimization problem, in this case over convex, lipschitz functions. (The restriction to a spherical domain is not really a restriction, you can just change variables for any bounde…

Sometimes I read a comment on HN that is so advanced that it's just as readable to me as Greek. Love reading it just to see someone work though!

It took me a while to understand a lot of these math concepts.

Turns out people doing Engineering research are using a very small but powerful bag of tricks from a handful of few famous Mathematicians. The concepts are named after them!

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#362

Post by hodgehog11 »

_alternator_ wrote: post #115
hodgehog11 wrote: post #92

Earlier quoted context omitted.

Very confused by this comment. The older (poorer) parts of the ML literature focus on models with convex and (gradient-)Lipschitz objectives, but that's not representative of reality, not even close. Modern objectives for AI models are famously nonconvex (catastrophically, from the point of view of classical optimisation theory), and that's where the interesting research is.

I'd push back on this. Most of the core optimization techniques (eg, ADAM, stochastic gradient descent) are straight out of the convex optimization literature. Generally you need to use optimizers that work well on convex objectives because near minimizers, functions tend to be convex. (Proof by contradiction: a non-convex point has a strict descent direction.) The fact that neural networks are highly nonconvex has e…

No, I have to push back as well, sorry. It takes a very long time to get to the "near-minimizer" stage when training a neural network, and in practice, you never get there (see neural scaling law regimes). What you are saying is the viewpoint from 6-7 years ago. Things have changed.

The reasons why optimizers work well for neural networks in their highly nonconvex landscapes has absolutely nothing to do with their performance in convex landscapes. If that were true, everyone would be using Newton-CG. These optimizers were born in the convex optimization literature as a consequence of the genetic optimization nature of incremental publication (and because that was all we had), but their modern study is through the lens of implicit regularization (their preferences for certain minima) and their stepwise vs. continuous rates for feature learning in multilayer models.

This is completely new theory by the way, and requires painful reinvention of the field. It does not stand on the shoulders of convex optimization. The nonconvex setting is assuredly not a perturbation of the convex setting, and those that do continue to work on deep learning optimization from the convex optimization perspective are well behind the times.

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#363

Post by light_hue_1 »

hodgehog11 wrote: post #362
_alternator_ wrote: post #115

Earlier quoted context omitted.

I'd push back on this. Most of the core optimization techniques (eg, ADAM, stochastic gradient descent) are straight out of the convex optimization literature. Generally you need to use optimizers that work well on convex objectives because near minimizers, functions tend to be convex. (Proof by contradiction: a non-convex point has a strict descent direction.) The fact that neural networks are highly nonconvex has e…

No, I have to push back as well, sorry. It takes a very long time to get to the "near-minimizer" stage when training a neural network, and in practice, you never get there (see neural scaling law regimes). What you are saying is the viewpoint from 6-7 years ago. Things have changed. The reasons why optimizers work well for neural networks in their highly nonconvex landscapes has absolutely nothing to do with their pe…

If people want to learn more, at ICML (one of the top AI/ML conferences) this year there was a whole tutorial on this topic: https://www.cs.ubc.ca/~schmidtm/Documents/2026_ICML_Tutorial...

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#364

Post by big-chungus4 »

_alternator_ wrote: post #115
hodgehog11 wrote: post #92

Earlier quoted context omitted.

Very confused by this comment. The older (poorer) parts of the ML literature focus on models with convex and (gradient-)Lipschitz objectives, but that's not representative of reality, not even close. Modern objectives for AI models are famously nonconvex (catastrophically, from the point of view of classical optimisation theory), and that's where the interesting research is.

I'd push back on this. Most of the core optimization techniques (eg, ADAM, stochastic gradient descent) are straight out of the convex optimization literature. Generally you need to use optimizers that work well on convex objectives because near minimizers, functions tend to be convex. (Proof by contradiction: a non-convex point has a strict descent direction.) The fact that neural networks are highly nonconvex has e…

I'd say it's going to be very hard to come up with a method that works on general nonconvex functions while not working on convex functions

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#365

Post by fooker »

_alternator_ wrote: post #76

I know a bit about this field. This conjecture reads as somewhat more niche than the cyclic double cover conjecture recently proved by OpenAI, but nevertheless represents a real contribution. You want to know how long it takes to solve an optimization problem, in this case over convex, lipschitz functions. (The restriction to a spherical domain is not really a restriction, you can just change variables for any bounde…

Any implications for discrete/integer optimization?

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#366

Post by fooker »

spicyusername wrote: post #297
xeromal wrote: post #165

Earlier quoted context omitted.

Sometimes I read a comment on HN that is so advanced that it's just as readable to me as Greek. Love reading it just to see someone work though!

Not to diminish the comment, but most things are not as complex as they sound when phrased in everyday language or sound much more complex than they are when phrased in technical language. Technical language is a tool that allows insiders to say less and refer to more, and to be specific, but it's just a tool. Most things can be described in accessible ways. I think you'd be surprised at what you could understand and…

Saying things colloquially gives everyone an intuition about it.

You'd think intuition is great, but no, about half of the important things are counter intuitive.

That makes intuitive thinking about complex technical topics you don't know enough about worse than useless.

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#368

Post by goosejuice »

alexpotato wrote: post #304
sigmarule wrote: post #197

Earlier quoted context omitted.

That era is now! (Still)

Paintball had a peak in the mid-2000s (I would say 2006) due to a combination of: - Millenials who were kids of the baby boomers being in their late teens early 20s - Disposable income due to the real estate bubble / positive consumer sentiment It dropped off a lot after the 2008 GFC though. BUT A lot of those kids playing in the mid 2000s are now parents of ~10 year olds so apparently there is a bit of a resurgence…

There was even a hl mod with an esports league.

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#369

Post by bananaflag »

paytonjjones wrote: post #191
spwa4 wrote: post #100

The problem is that we're going to have another deepseek moment when someone uses GLM or Kimi K3 to do this.

What was the first DeepSeek moment? (genuine question, I'm out of the loop on what you mean)

January 2025, release of DeepSeek R1, the first open reasoning model. There was a lot of panic then that it was done with very few resources.

Re: GPT-5.6 used a prompt to close a 30-year gap in convex optimization

#370

Post by andy12_ »

YeGoblynQueenne wrote: post #327
andy12_ wrote: post #198

Earlier quoted context omitted.

> but it is worth noting that this wasn't a matter of "ChatGPT, solve this unsolved problem. Make no mistakes." It wasn't the case for this, but when OpenAI disproved the Unit Distance Conjecture, it was really done autonomously by an automated AI pipeline with a completely AI-generated prompt. No human expertise required at all in the process (well, except for the final human verification).

Without human verification, an LLM can generate correct or incorrect proofs but it can't tell the difference. A human is necessary to be able to tell one from the other. Saying that's a solution "done autonomously by an automated AI pipeline" is like saying that a self driving car that can only take you to the nearest train station after which you have to ride the rain to where you're going is "autonomously" driving…

The automated AI pipeline also had an automatic grading model to try to reduce false positives.

But anyway, my point was that in that case the prompt involved was indeed pretty much "hey, ChatGPT, solve an unsolved problem, thanks."

Post reply on HN