I think the argument for the pelican is if it cannot do a simple thing perfectly. A more complex thing is just a stacking a small errors.
Karpathy’s Pelican
391–400 of 461 posts
Re: Karpathy’s Pelican
#392This is equally bad as a pelican test. LLMs should be tested in the same way people should be tested for a job interview (but often aren’t) - with tasks RELEVANT to usage. So you don’t just randomly pick some random thing to make the LLM randomly do (like many job interviewers do). You start with clear statements about real world usage scenarios. THEN you come up with tests that give insight to how well the LLM/hob s…
You might be onto something there, except that people do use LLMs for these exact use cases.
Re: Karpathy’s Pelican
#393Earlier quoted context omitted.
I don't know I think it is charming in a way that is lacking in nearly everything else an LLM tries to do creatively. I have always preferred the result of getting an LLM to draw an svg or make a procedural animation like this to the uncanny hyper-realistic result of diffusion image/video generation.
To help explain AI to my elderly mother, I said it is like an alien intelligence on a planet too far away to directly observe Earth, and everything it knows about humanity and our world it learned from reading just about every book and website. You can ask it a question or to do some work and it can often give a useful response but it can never directly judge its accuracy if it relates to the physical world, it has o…
That's a nice metaphor for your mother, i like it. I would just touch on one small part, which is that it has more than our judgement though. It has access to laws of physics, to formulas, to all our current scientific knowledge which we have shown to be correct by interacting with the physical world. It can create an (incomplete) model of the world based on verified theories already. Through deduction and reasoning alone it can get pretty far. After all, there's many scientific theories we created long in advance before we actually proved them to be correct through interaction with the physical world. So even before they were proved to be correct, they were already correct. Same can be applied to what LLM can infer through reasoning alone.
Anyway, just a small point which probably still wouldn't change the nice metaphor your made.
Re: Karpathy’s Pelican
#394Earlier quoted context omitted.
Yet no LLM can actually do it. It's quite surprising actually.
What does this even test for? Can I use LLMs to directly generate machine code to replace my compiler? Or maybe I can use LLMs as a bare metal OS / scheduler to replace my machine's operating system and scheduler? It makes zero sense to test for that. Not only that this so-called "benchmark" isn't economically useful, but that it tests for the sake of testing; and for attention. To end this obsession with generating…
It tests for the "I" in "AI".
Re: Karpathy’s Pelican
#395A lot of people are posting here about how bad the end product is, but that is kind of the point. Models have moved beyond generating images to a new kind of benchmark that better exposes understanding of the physical world, and we can use benchmarks like this to measure future progress. (Of course, it will have to be a qualitative/subjective measurement.)
Will Smith spaghetti was garbage a couple of years ago and now AI videos are becoming close to indistinguishable from real videos in many cases. Spaghetti 2026: https://x.com/dreamingtulpa/status/2083304533829066873 https://xcancel.com/dreamingtulpa/status/2083304533829066873
Wiki page for people like me who never heard of this test: https://en.wikipedia.org/wiki/Will_Smith_Eating_Spaghetti_te...
Re: Karpathy’s Pelican
#396Earlier quoted context omitted.
> touching the limit of what one can reasonably expect If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that. And Elon is already propagating the age of abundance where money won't exist anymore, right before calling the interviewing journalist dishonest and deservedly losing public trust. Smh my head.
> If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that. What knowledge workers do you know that have excellent drawing skills? I worked in a design agency and for a couple of years, each week me and a few other people would attempt to sketch a member of our group: one person would be the model and sit still, and everyone els…
It is a proxy for intelligence. And as long as that metric is not gamed (which it surprisingly doesn't appear to be yet), the drawing skill of sth is a quite reasonable test.
It surprises me how many people in this community don't get this. Obviously, most prompts thrown into an AI chatbot/interface are about something no knowledge worker would ever have to deal with. That doesn't disqualify them as a tool for measuring progress of the models.
Re: Karpathy’s Pelican
#397> Last thought is that the domain of worlds/games exposes a weakness in LLMs: they can't easily audit their work because they aren't able to efficiently and natively perceive videos or play games within them.
Again comes back to the point of verifiable rewards. The moment you take it away from LLMs they just stop being as good
Re: Karpathy’s Pelican
#398Earlier quoted context omitted.
I think it's interesting to see them visibly struggling to improve. Claude pelicans aren't much better today than they where 18 months.
Rendering 3d worlds has hugely improved though.
they are quite good also at placement and creating scenes etc. I had one implement a cascading shadow system in vulkan/glfw and just fed it back screenshots with peter pannin and acne spots etc.
it made a huge monstrosity first, 1500+ lines of shader code. Once it was happy with the result (it looked pretty good, almost blenders gamerenderer) it cleaned it up and a lot of debug code was removed. shrank it down to about 350 lines and made it much more readable.
this was something i didnt expect it to be able to do. create shaders, look at screenshots, fix em iteratively like that. multi modal debugging.
Re: Karpathy’s Pelican
#399It seems pretty clear that Anthropic models have been specifically trained to be good at generating three.js (JavaScript 3-D Graphics) code, so given current state of AI code generation in general, I don't find three.js models/animations as indicative of anything other than the model's ability to write three.js code. When Fable was first released the day-1 demos of it on Twitter (presumably from people who were given…
I don't find three.js models/animations as indicative of anything other than the model's ability to write three.js code. This weekend I've been converting a game from three.js to ogl.js in order to see if I can optimise the time-to-interactive loading time. I took the three.js driven page weight from about 600KB (500KB being three.js) to about 50KB, and reduced the loading time from multiple seconds on a 4G mobile co…
The UI is the equivalent of "You're absolutely right!". It's a big turnoff at least to me. That toast is such an AI thing.
Re: Karpathy’s Pelican
#400I've come across the same issue at home with my qwen3.6 agent in Pi. It has to take screenshots of a 3D scene and look at the screenshots to see what is happening. It can even produce a video for me with ffmpeg, but it can't watch the video or see the render output directly, only take screenshots for review.