This paper is being published at ICLR 2026 (top AI conference), and was selected as one of three outstanding papers.
Transformers are inherently succinct
openreview.net
Transformers are inherently succinct
1–10 of 46 posts
Re: Transformers are inherently succinct
#2This paper is being published at ICLR 2026 (top AI conference), and was selected as one of three outstanding papers.
Re: Transformers are inherently succinct
#3Discussed (a bit) here:
Transformers Are Inherently Succinct (2025) - https://news.ycombinator.com/item?id=48014197 - May 2026 (9 comments)
Re: Transformers are inherently succinct
#4This paper is being published at ICLR 2026 (top AI conference), and was selected as one of three outstanding papers.
(We'll add that bit to the toptext as well. Thanks!)
Re: Transformers are inherently succinct
#5I had no idea that LLMs (or the transformer architecture) were within reach of complexity theory. But if transformers "can be" exponentially more succinct than RNNs, doesn't that mean we're approaching optimality?
Re: Transformers are inherently succinct
#6What about the other direction? What languages are expressible w/ RNNs & LTLs that require exponential blowup for transformers?
Re: Transformers are inherently succinct
#7Paper went over my head but is this in any way related to my experience of Claude Opus 4.8 using increasingly terse language with very short, overloaded words? Lately I've been having trouble parsing the things it writes about my own code, it's using the kind of compressed language that you see typically in git commit message subject lines but relentless, always on.
Re: Transformers are inherently succinct
#8Paper went over my head but is this in any way related to my experience of Claude Opus 4.8 using increasingly terse language with very short, overloaded words? Lately I've been having trouble parsing the things it writes about my own code, it's using the kind of compressed language that you see typically in git commit message subject lines but relentless, always on.
No, this is in the same ballpark as ideas like big-O notation. The paper is saying that transformers can recognize a language with exponentially fewer symbols than other kinds of systems, i.e. they're more succinct.
It's exactly as related to real models as computer science is to real computers.
Re: Transformers are inherently succinct
#9The last line of the abstract has the most important takeaway.
> As a consequence of this succinctness, we show that basic verification problems for transformers, such as emptiness and equivalence, are provably intractable: specifically, EXPSPACE-complete.
If you were hoping to formally prove the correctness of a large transformer, it turns out that you're going to need an exponentially larger amount of space to do your verification, more than you could possibly afford.
Re: Transformers are inherently succinct
#10Paper went over my head but is this in any way related to my experience of Claude Opus 4.8 using increasingly terse language with very short, overloaded words? Lately I've been having trouble parsing the things it writes about my own code, it's using the kind of compressed language that you see typically in git commit message subject lines but relentless, always on.
Not just me then.