Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Karpathy’s Pelican

twitter.com

Post reply on HN
Print view View original

261–270 of 461 posts

Re: Karpathy’s Pelican

#261

Post by nightshift1 »

lowbloodsugar wrote: post #153
qrios wrote: post #89

Earlier quoted context omitted.

I'm sure this will become the standard. And plastic is an excellent analogy. Maybe we can take it a step further and compare it to on-demand 3D printing. Why would anyone still use off-the-shelf software when they can have a system that has access to all data, can transform it into any form, and can export it in any format? After years of thinking that I needed to develop a decent movie management system for my own f…

This. This is why the burst of posts on HN of “I made this useful tool/crate/application” were just so sad. The old model was getting what the kids now call aura by developing useful open source products: products where it’s far easier for someone to consume the product than write it themselves. We had people posting things as if that model still existed. Dude, you wrote it with an LLM! Posting them (here) is not onl…

but are you willing to spend the time or tokens to build it again for yourself ?

Re: Karpathy’s Pelican

#262

Post by wrxd »

criddell wrote: post #35
blitzar wrote: post #28

I think the pelican test is better.

I'd like to see tests of things the current AIs are bad at, like drive a car. Or maybe take instruction to complete some novel activity to test how well they can learn.

Would you say that the generated video is good?

Better than I would have expected? Sure. Impressive that it got that far? Definitely. But is the end result good?

Re: Karpathy’s Pelican

#263

Post by emp17344 »

anigbrowl wrote: post #207
dundarious wrote: post #51

I can forgive the modeling being godawful jank (windows floating in the air, disconnected from the house). But I expected it to have a better understanding of the text. Instead, we have Bilbo's "disappearance" interpreted as him magically transporting or cloaking, and similarly for his reappearance.

But this only seems wrong to you because you're familiar with the prior context. When there's only a single paragraph to work from, and it's a drily humorous text, why not employ comic literalism and lean into the perplexity with which his neighbors viewed him?

This is a rationalization. The simpler explanation is that the model failed to properly depict the passage. As the AI booster crowd would say, cope.

Re: Karpathy’s Pelican

#264

Post by meander_water »

recursive wrote: post #256
meander_water wrote: post #253

Earlier quoted context omitted.

Can someone explain what the pelican on a bicycle tests exactly? And why is it so important? I've never understood how it could translate to a useful task in real life.

If your school mascot is a pelican and you need to make a flyer for the upcoming bike safety event, then this precise thing becomes useful. Most things that are useful in the real world seem not useful out of context.

Sure, but then you would just use an image generation or multimodal model to generate that image. I don't think you'd want a weird looking svg.

Re: Karpathy’s Pelican

#265

Post by enos_feedler »

Morromist wrote: post #238
jonas21 wrote: post #209

Earlier quoted context omitted.

I don't think he's claiming it's been exhausted. It's just that things have progressed to a point where people are arguing over the finer points of which pelican looks better -- which is often a matter of taste, and an indication that we've hit the knee in benchmark where models are no longer failing in obviously awful ways.

I haven't really seen evidence that any ai can reliably draw a pelican riding a bicycle. Not if you look at the image long enough to take it in. Even the best ones have something wrong with them. Not a matter of taste but a matter of having both legs peddling on the viewer's side of the bicycle or having two beaks. I'm actually beginning to wonder if some people who ignore these things have a different, somewhat less…

AI has deeply changed the way I think, feel and act around a computer. In the same way that dialing into the internet changed things for me. Since using ChatGPT the first time until now I have never cared once to look at these pelicans on bikes people seem to get hung up about. It could never have been a thing and nothing would change. See the forest through the trees.

Re: Karpathy’s Pelican

#266

Post by sixtyj »

user43928 wrote: post #240
sixtyj wrote: post #196

Earlier quoted context omitted.

Have you seen pelicans in Simon Willison’s tests? It is still not a pelican on bike I would like to publish :) Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time. It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see…

Do you expect the SVG to emulate a hand-drawn picture, become more realistic, or just a more detailed illustration? As for the often quoted issues with the bike's frame or problem with the steering column, I can't really tell, I am no bike expert. I can instead judge how poor of a job it is doing with a LOTR rendition in Three.js, so that seems like a better benchmark.

A general benchmark (even Simon mentioned that it was meant as fun at the beginning) should be quick and easy to run, since we can expect that more people will want to try it out. That’s why I’m more like “team Pelican on a Bike”… :) cheers

Re: Karpathy’s Pelican

#267

Post by recursive »

meander_water wrote: post #264
recursive wrote: post #256

Earlier quoted context omitted.

If your school mascot is a pelican and you need to make a flyer for the upcoming bike safety event, then this precise thing becomes useful. Most things that are useful in the real world seem not useful out of context.

Sure, but then you would just use an image generation or multimodal model to generate that image. I don't think you'd want a weird looking svg.

Exactly. I would want a good looking svg. That's the evaluation.

Re: Karpathy’s Pelican

#268

Post by bredren »

9dev wrote: post #245
bredren wrote: post #8

I worked with an LLM to build a ~3D animation of the Back to the Future delorean Time Machine as a way to spice up the hero on a docs page. That took a fair amount of custom tuning and I had to create a tuning view to get some of the behaviors right. But it was enough fun that I generalized it to take in ~any scene description from a film. It goes out and gets more detailed descriptions and film stills if available b…

Ha, that's fun! I've built half your product for myself as a personal, native MacOS app, with a slightly different direction, but the same core idea. I might go do a little scavenging for ideas in your docs :-)

Please feel free to do so! I have a source available copy of the self-hosted sync server up on github here: https://github.com/PeterPym/contextify-cloud-self-hosted

If you have time to try the product as it stands, I would definitely appreciate your feedback in particular, and would be interested in checking out what you've built so far for yourself.

I will publish the API to the database as well.

Re: Karpathy’s Pelican

#270

Post by w4yai »

Morromist wrote: post #238
jonas21 wrote: post #209

Earlier quoted context omitted.

I don't think he's claiming it's been exhausted. It's just that things have progressed to a point where people are arguing over the finer points of which pelican looks better -- which is often a matter of taste, and an indication that we've hit the knee in benchmark where models are no longer failing in obviously awful ways.

I haven't really seen evidence that any ai can reliably draw a pelican riding a bicycle. Not if you look at the image long enough to take it in. Even the best ones have something wrong with them. Not a matter of taste but a matter of having both legs peddling on the viewer's side of the bicycle or having two beaks. I'm actually beginning to wonder if some people who ignore these things have a different, somewhat less…

> I haven't really seen evidence that any ai can reliably draw a pelican riding a bicycle

Please remember, we've started from there :

https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/

When it started, it was clear what LLM would stand out, its style, etc. Nowadays, the pelicans look similar, the difference is in details and sometimes hard to catch. Sure, the task is not completed perfectly, but that's not the point. It was supposed to be a benchmark to quickly benchmark a LLM against others.

Post reply on HN