Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Karpathy’s Pelican

twitter.com

Post reply on HN
Print view View original

291–300 of 461 posts

Re: Karpathy’s Pelican

#291

Post by RugnirViking »

beepbooptheory wrote: post #215
fasterik wrote: post #190

Earlier quoted context omitted.

Because the degree of realism in a video is determined by implicit knowledge of concepts like "in front of", "behind", "next to", "inside", "outside", "between", "occluded by", etc. as well as distances, angles, and the relative size of objects when viewed from a perspective.

I guess I just don't have an intuition for why that is different from what it does for non-"spatial" things. Like the fact that it works is still because it produced the code it did token by token. Whether its dealing with, e.g., " beside the rock" or " in the array", it's doing the same kind of inferential activity.

> I just don't have an intuition for why that is different from what it does for non-"spatial" things

A big reason people separate this out is because it was only a short time ago that AI models were noticably and uniquely bad at this. I don't say this to be like "oh so imagine where theyll be in x amount of time", rather that this thing that was once a serious limitation of the technology is slowly being compensated for by larger and better trained models

Re: Karpathy’s Pelican

#293

Post by Wowfunhappy »

skygazer wrote: post #286
Wowfunhappy wrote: post #283

Earlier quoted context omitted.

> I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle". Oh come now. I am extremely confident that if I hired a professional artist to draw a picture of a pelican riding a bicycle, I would get something inarguably much better than what today's best coding LLMs can produce.

At that rate you could use an image model, which was designed for the task. Thats the absurdity of this test. It’s often a text only generation model that has never seen a pelican, coerced into creating an xml graphical representation that a human might recognize. That it does anything passable is already astounding.

I agree, it's hard! That's why it's still a good benchmark.

Re: Karpathy’s Pelican

#294

Post by techpression »

skygazer wrote: post #286
Wowfunhappy wrote: post #283

Earlier quoted context omitted.

> I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle". Oh come now. I am extremely confident that if I hired a professional artist to draw a picture of a pelican riding a bicycle, I would get something inarguably much better than what today's best coding LLMs can produce.

At that rate you could use an image model, which was designed for the task. Thats the absurdity of this test. It’s often a text only generation model that has never seen a pelican, coerced into creating an xml graphical representation that a human might recognize. That it does anything passable is already astounding.

It probably has a few million inputs on how a pelican and bicycle looks, not to mention the amount of data on how to create SVG’s. Ask it to create a relaxing spa website and it will, even though it has never seen a spa.

Re: Karpathy’s Pelican

#295

Post by Demiurge »

enos_feedler wrote: post #265
Morromist wrote: post #238

Earlier quoted context omitted.

I haven't really seen evidence that any ai can reliably draw a pelican riding a bicycle. Not if you look at the image long enough to take it in. Even the best ones have something wrong with them. Not a matter of taste but a matter of having both legs peddling on the viewer's side of the bicycle or having two beaks. I'm actually beginning to wonder if some people who ignore these things have a different, somewhat less…

AI has deeply changed the way I think, feel and act around a computer. In the same way that dialing into the internet changed things for me. Since using ChatGPT the first time until now I have never cared once to look at these pelicans on bikes people seem to get hung up about. It could never have been a thing and nothing would change. See the forest through the trees.

What you’re saying is that you’re not interested in benchmarks. But then you go a step further and state that this particular benchmark is entirely inconsequential. That’s like telling you that if you didn’t exist, nothing would change. Even if that were true, it would still be an insensitive and rude thing to say, wouldn’t it?

Re: Karpathy’s Pelican

#296

Post by reaperducer »

nemothekid wrote: post #289
reaperducer wrote: post #279

Earlier quoted context omitted.

Sure, the task is not completed perfectly, but that's not the point. Isn't it? If the computer can't do it better than a human being, then what's the point? Being wrong at scale is not better than being right.

> If the computer can't do it better than a human being, then what's the point? Because the benchmark wasn't testing "can an LLM draw a pelican like a human". The original article was testing the relative capabilities between LLMs. Now that LLMs can all draw pelicans all similarly, the test is less interesting as a comparative benchmark.

Trillions of dollars spent. Trillions of gigawatts consumed. And people still celebrate "Yay! We're less wrong than the other guys!"

This is what the tech industry has become?

Less of a failure is still failure.

Re: Karpathy’s Pelican

#297

Post by peterbower »

matsemann wrote: post #41

I'm pretty tired of the "Y made this game in Z tokens" all over the internet last week. They look impressive, and it's cool that it's even possible, but they're useless as games. None of them are any fun. They're like the most boring variant of basic controllers you can imagine. None have any cool mechanics. None have any tweaks made from hours and hours of testing. All have the same cel-shader.

I agree - they have not been good games. However, the fact that it is able to build and work with shaders effectively is promising - massive time saver and lowers the barrier to entry for people who need these capabilities without double majoring in advanced mathematics and computer science/game design.

One would hope it accelerates AR/VR - if decent studios finally get behind it.

Re: Karpathy’s Pelican

#298

Post by adriand »

teiferer wrote: post #255
BobbyJo wrote: post #228

Earlier quoted context omitted.

I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle". When you aren't sure if an LLM can write an svg well, or that it will be able to form a pelican shape, or animate a bicycle, it's a good test. After that, it's all judgement: how detailed should the pelican be? pelicans are the wrong shape for…

> touching the limit of what one can reasonably expect If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that. And Elon is already propagating the age of abundance where money won't exist anymore, right before calling the interviewing journalist dishonest and deservedly losing public trust. Smh my head.

> If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that.

What knowledge workers do you know that have excellent drawing skills? I worked in a design agency and for a couple of years, each week me and a few other people would attempt to sketch a member of our group: one person would be the model and sit still, and everyone else would draw her/him.

Let me tell you, if producing a convincing portrait was a prerequisite for being a knowledge worker, there would be 99% fewer knowledge workers.

Re: Karpathy’s Pelican

#299

Post by tayo42 »

HarHarVeryFunny wrote: post #244
jcims wrote: post #202

I’d like to see a human one shot a pelican on a bicycle in raw svg.

It's been shown in tests that most humans can not from memory draw a functional bicycle given pen and paper. Everyone knows roughly what a bicycle looks like - wheels, frame, seat, peddles, handlebars etc, but the details of the frame and exactly how the other parts connect to it throws people off. It seems people memorize the "concept" of a frame, but not the specifics. Try it without cheating, then google a picture…

People don't memorize anything like that.thats why art is done with reference images

Re: Karpathy’s Pelican

#300

Post by meander_water »

RugnirViking wrote: post #273
meander_water wrote: post #253

Earlier quoted context omitted.

Can someone explain what the pelican on a bicycle tests exactly? And why is it so important? I've never understood how it could translate to a useful task in real life.

On the most basic level, its asking the ai to generate valid svg code for a picture of a pelican riding a bicycle, as a way of checking its intelligence. Popularized (invented?) by simonw, its been used as part of his reviews of new models as they come out since oct 2025. It used to be a very difficult task for models, see [2,3,4] it cuts across several tasks that AI used to be very bad at, but now has improved quite…

This is what I was looking for, thanks!
Post reply on HN