Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra, looped transformers, and hidden reasoning

magazine.sebastianraschka.com

Post reply on HN
Print view View original

91–100 of 156 posts

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#91

Post by nonethewiser »

bradly wrote: post #79
dooglius wrote: post #72

Earlier quoted context omitted.

Do you have hard evidence of this assertion?

There is a toot from an Open AI person a couple days ago saying they are "pulling all the levers" because of capacity issues. I have no idea what the heck the person is talking about, but I'm guessing there are consequence for those levers. > "Demand for Astra is really unprecedented. We're pulling all the levers possible to sustain the demand, but I've not seen anything like it until now and we went through very ste…

Depends on the nature of the levers

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#92

Post by marcus_cemes »

fragmede wrote: post #86
simlevesque wrote: post #78

Earlier quoted context omitted.

We can't have hard evidence. It's a SaaS and they own the code and the machine it runs on. So it may be a widespread hallucination. But there's no evidence of that either.

We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.

You would need a significant sample size to make any sort of conclusion from such a probabilistic process. Then there's the issue of how you would actually grade/compare.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#93

Post by ArvidSu »

fragmede wrote: post #86
simlevesque wrote: post #78

Earlier quoted context omitted.

We can't have hard evidence. It's a SaaS and they own the code and the machine it runs on. So it may be a widespread hallucination. But there's no evidence of that either.

We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.

You only need to come up with a catchy "SomethingBench" name, post it on reddit/x and now you're an ai sage. Not to disparage the launch/after comparison though, I'd genuinely enjoy a data point like that

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#94

Post by frunkp »

When I saw "hidden reasoning", it reminded me of diffusion models: generating a block spans many steps (with remasking), which hide the reasoning that led to the block.

I had not heard of looped transformers, but the engineering behind the number of loops per token / halting feels like trying to apply a diffusion process to a transformer while keeping the auto-regressive feature.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#95

Post by luckydata »

fragmede wrote: post #86
simlevesque wrote: post #78

Earlier quoted context omitted.

We can't have hard evidence. It's a SaaS and they own the code and the machine it runs on. So it may be a widespread hallucination. But there's no evidence of that either.

We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.

someone already does that https://aistupidlevel.info/

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#96

Post by Caracas288 »

selectodude wrote: post #84
NineStarPoint wrote: post #69

Earlier quoted context omitted.

Yeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.

NVFP4 would buy them a huge increase in capacity but I think it would be noticeable.

Why doesn't someone just try to measure this next time!?

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#98

Post by torginus »

boredatoms wrote: post #55
baby wrote: post #45

Earlier quoted context omitted.

You think they introduce stronger quantization after a few days?

For sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain

Some people here have remarked previously that while reduced precision doesn't show up in quick prompts, it does severely impact these models' ability to perform long running tasks - to the point that running these big models with severe quantization might be counterproductive as smaller but less quantized ones perform better.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#99

Post by brausepulver »

andai wrote: post #77

> So, the whole idea here is that we increase the effective depth from 22 to 44 block applications without adding another set of transformer weights. From what I gathered, LLM inference is bottlenecked on memory, right? Which implies there's "spare" compute we haven't been using? Does reusing the weights like this allow us to utilize it? (Do more math per unit of memory?)

You need to separate memory capacity and bandwidth. Looping decreases memory capacity/FLOP but not bytes loaded/FLOP, since weights need to be loaded again for the 2nd pass. Plus (depending on the method used) capacity required for KV will be that of the equivalent unlooped model (44 blocks) and KV is typically larger than weights at long context.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#100

Post by chaimtweiss »

luckydata wrote: post #95
fragmede wrote: post #86

Earlier quoted context omitted.

We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.

someone already does that https://aistupidlevel.info/

It's actually a extremely cool site, and fascinating to view the results off the AI bots i use.
Post reply on HN