Kimi K3, and what we can still learn from the pelican benchmark

I think they're less and less advertised as true generalists these days, as they pivot to profits that obviously lie (for the time being) first and foremost in agentic coding. It's no longer unusual to see regressions in terms of more stiff prose due to the strong tuning towards coding, or how they structure their response. And prose is a LLM's home turf! Instead, progress in agentic coding capability is usually the headline feature, the headline benchmark, etc etc. At least looking at Anthropic, Google, OpenAI. There are of course other LLM's.

So then add a dash of cybersecurity and medical use and that's basically it. No "closer to AGI" advertising. I'd say the 2026 development has in fact been the opposite; optimizing AI for niches where there is most potential for profits and that your description died in circa GPT-5 era.

In fact, this problem (for this test) is also stated by the pelican test author:

"The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.

So don’t go using pelicans to compare models!"

LLMs are, fundamentally, generalist AIs. Marketing or no marketing - it's just what they are. How they're trained, how they perform, what they're best at.

Empirically, they have something very much alike to the human "g factor" - a shared pool of "general intelligence" that all tasks benefit from.

When a "make it bigger, train it harder" upgrade like Kimi K3 or Mythos 5 drops, the performance rises on every metric. Not just the "headline benchmarks" like Mythos and coding/cybersecurity, but also things like literary analysis - which has nearly zero economic value, and isn't commonly post-trained or benchmarked for. And companies keep encountering things like "our carefully trained specialist model with lots of in-domain training on expensive closed datasets just got leapfrogged on our internal benchmarks by a next gen off the shelf generalist".

You can go hard on benchmarkmaxxing post-training, and you can burn millions of GPU-hours on coding RLVR. But, by the very nature of LLMs, a lot of the performance gains in flagship models are broad and domain-inspecific.

"Stiff prose" is more of a "style" thing than a "capability" thing. No one cares about how good an AI is at things like long form creative writing, because that's the opposite of a profitable field. All of LLM behavior is routed through text, so it's very easy to perturb "writing style" by some training elsewhere. Regression evaluation is hard. And the writing-specific post-training LLMs get is usually just cheap RLAF, with all the usual RLAF degeneracy.

Thus, we get the "default styles" that suck from a "creative writing" standpoint. A lot of that is just "what sounded good to the previous generation of LLMs" - and, unlike human readers, LLM evaluators don't get bored from seeing the same cliches repeated 9000 times across 9000 different instances of generated text. Humans tend to update over time from "this sound cool and punchy" to "this is generic AI slop", but RLAF evaluators stay at step 1. What little human-guided optimization this gets is aimed at "copywriting, marketing blurbs, punchy short-form" - and it shows.

You can do a lot there with some aggressive prompting, but the default writing styles suck, and I frankly don't expect that to change soon. No one cares enough to change it.

Pelicans? Used to be a decent proxy for "general model capabilities that no one would benchmaxx for" - a way to probe for that elusive "LLM g factor". Now that it's a known metric, it's very gameable. But it was pretty solid while it was novel and obscure.

The models are already brilliant at that. My own todo app generates 128x128 pixel art icons for my todo items. They are mind blowingly creative and funny.

It is a simple prompt that packs a lot:

* operates an absurd prompt

* involves SVG coding knowledge, generates a source code artifact

* involves world knowledge (what is a pelican? What is a bicycle? What does each do?” How are each constructed?”)

* when rendered, the coding artifact expresses an image that makes sense to us perceptually, including color and spatial relationships

* different models and settings have different output so it can be used as an evaluation scheme

That said I wouldn’t choose a model based on this! Just like some brain teaser shouldn’t determine employment eligibility.

That's the joke.