Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Karpathy’s Pelican

twitter.com

Post reply on HN
Print view View original

351–360 of 461 posts

Re: Karpathy’s Pelican

#351

Post by YmiYugy »

pj_mukh wrote: post #306
Wowfunhappy wrote: post #283

Earlier quoted context omitted.

> I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle". Oh come now. I am extremely confident that if I hired a professional artist to draw a picture of a pelican riding a bicycle, I would get something inarguably much better than what today's best coding LLMs can produce.

> if I hired a professional artist to draw a picture of a pelican riding a bicycle, I think AI folks have done a terrible job of communicating this, but replacing a professional simply isn't the point. The point is to serve all the situations where people would've never considered hiring a professional, and where perfection or artistic merit isn't the point (say a personal throwaway recreation of an LOTR world). And…

IMHO the output is bad enough that I can't imagine a use case for illustrations of this kind.

Re: Karpathy’s Pelican

#353

Post by consumer451 »

vanjajaja1 wrote: post #350
consumer451 wrote: post #339

The difference between this and Simon's pelican is that with Simon, I get the prompt. Last I checked, I did not see the prompt for this really cool thing, so it is not reproducible. Did I miss the prompt somewhere?

he said the prompt was the first paragraph of LoTR, but he didn't mention a preamble this guy seems to have taken that idea and got something similar/better, so likely the prompt isn't too special https://x.com/Izkimar/status/2083819741643178208?s=20

That was far worse as far as visual story-telling. Anthropic wins, which is to be expected against whatever that product is.

In either case, I still don't see how I could reproduce this to test against various models, which is the entire point of Simon's pelican.

Re: Karpathy’s Pelican

#354

Post by cyanregiment »

sixtyj wrote: post #196
maxutility wrote: post #109

Earlier quoted context omitted.

Agree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting. A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and firs…

Have you seen pelicans in Simon Willison’s tests? It is still not a pelican on bike I would like to publish :) Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time. It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see…

> useless 3D worlds that are just a bragging

No, this demo is the useless 3D world, and you're bragging.

A real game would have a lot more immersive of a world, and you wouldn't need to.

Re: Karpathy’s Pelican

#355

Post by weakfish »

Out of curiosity, I asked Opus 5 to do the same with the first ~1.5 pages of Neuromancer by William Gibson, which I thought might be a good test of interpretation from the model.

It refused to use the text verbatim because of copyright (ironic), but the output was interesting nonetheless.

https://claude.ai/public/artifacts/275dc3c2-7bd3-432b-94ff-d...

Re: Karpathy’s Pelican

#356

Post by CapsAdmin »

Something that bugs me about these new demos is that they benchmark the model and harness at the same time. The harness steer the model to expand the vague prompt, self correct with tools, etc.

I think it's useful when evaluating "model+intended harness", but I'm more interested in seeing raw model improvements than harness improvements.

Re: Karpathy’s Pelican

#358

Post by HarHarVeryFunny »

dannyw wrote: post #341
HarHarVeryFunny wrote: post #47

It seems pretty clear that Anthropic models have been specifically trained to be good at generating three.js (JavaScript 3-D Graphics) code, so given current state of AI code generation in general, I don't find three.js models/animations as indicative of anything other than the model's ability to write three.js code. When Fable was first released the day-1 demos of it on Twitter (presumably from people who were given…

Why do you think it's pretty clear? Anthropic models are excellent at working with Blender APIs, other industry standard 3D modelling programs and tasks, and game libraries that have nothing to do with Three.js. Three.js is a quite popular library; and browser-based apps are more easily sharable and more portable. So models having a preference for using it when unprompted doesn't suggest anything, just like how model…

It was very notable that most of the day-1 Fable demos were all three.js based, which seems too much to be a coincidence, especially since I expect anyone giving day-1 demos are basically insiders (early access or employees).

It seems they trained it to be good at it, then requested everyone to demo it.

Re: Karpathy’s Pelican

#359

Post by iDon »

I just went down a rabbit-hole, and happily I found the rabbit - an early example of this type of prompt (draw an unusual animal character in a vector graphics language). There is a paper and a 1 hour podcast resulting from a Microsoft evaluation of a pre-release of GPT4. One of the prompts (see pages 4,7,8 in the paper PDF) was to draw a unicorn in TikZ. (I'll leave to the historians the questions of whether this was the first, or whether Simon Willison may have been inspired by this). I remember hearing some of the podcast, and recalled the prompt about balancing on a nail, and the triangle forming the unicorn's tusk; this was clearly a big step beyond choosing the next word and blending images.

https://arxiv.org/abs/2303.12712

Re: Karpathy’s Pelican

#360

Post by bearjaws »

baron816 wrote: post #11

IMO, the area where AI is going to be most useful over the next couple years is in developing manufacturing processes top to bottom. Maybe a million token budget is too small, but something like "design me a sneaker and all the equipment to manufacture it autonomously".

LLMs are horrible at real world, physical movement of products and people.

I've seen them make time estimates that are physically impossible, expect medications to be in two places at once, etc.

Post reply on HN