Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Karpathy’s Pelican

twitter.com

Post reply on HN
Print view View original

191–200 of 461 posts

Re: Karpathy’s Pelican

#191

Post by barrenko »

fasterik wrote: post #168
barrenko wrote: post #124

This has started to feel a bit like the beginning of railroads and then the steampunk fiction of "let's just build railroads to everywhere". We don't need it and there's no use for it. As with painting, after a while there's nothing really new to paint, we genuinely need 0 new software. We need to fix our broken physical world, our social lives, our kids and what's left of our democracies. This software crap is done,…

Agreed that software isn't going to fix our institutions. But we need 0 new software? AlphaFold won its creators a Nobel prize in chemistry and solved a research problem that feeds into every area of biology, medical research, and drug discovery. We still have an untold number of unsolved problems related to human health, food production, energy production, infrastructure, transportation, education... The list is pra…

Agree for stuff like alphafold, for the likes of education, imho, it's been solved for at least a couple of centuries.

Textbook and blackboard > ipad.

Re: Karpathy’s Pelican

#194

Post by irthomasthomas »

maxutility wrote: post #109
jmugan wrote: post #93

A lot of people are posting here about how bad the end product is, but that is kind of the point. Models have moved beyond generating images to a new kind of benchmark that better exposes understanding of the physical world, and we can use benchmarks like this to measure future progress. (Of course, it will have to be a qualitative/subjective measurement.)

Agree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting. A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and firs…

I think it's interesting to see them visibly struggling to improve. Claude pelicans aren't much better today than they where 18 months.

Re: Karpathy’s Pelican

#195

Post by nozzlegear »

gegtik wrote: post #165
IsTom wrote: post #158

Earlier quoted context omitted.

Aren't they still bad at understanding how bicycle frame works? Especially the steering part?

Maybe that makes them human.. https://www.booooooom.com/2016/05/09/bicycles-built-based-on...

Humans aren't machines trained on the entire stolen corpus of human knowledge. We expect that a human will do poorly at arbitrary tasks they have no experience doing – especially drawing, which many (most?) humans aren't trained in at all. The same isn't true of AI, where its proponents, priests and proselytizers have spent time, energy and billions of dollars attempting to convince us it can do anything better than humans.

Re: Karpathy’s Pelican

#196

Post by sixtyj »

maxutility wrote: post #109
jmugan wrote: post #93

A lot of people are posting here about how bad the end product is, but that is kind of the point. Models have moved beyond generating images to a new kind of benchmark that better exposes understanding of the physical world, and we can use benchmarks like this to measure future progress. (Of course, it will have to be a qualitative/subjective measurement.)

Agree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting. A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and firs…

Have you seen pelicans in Simon Willison’s tests? It is still not a pelican on bike I would like to publish :)

Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time.

It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see Mark Zuckerberg’s ceased attempt with 3D VR…

Multiply it by thousands times as a lot of people have found out threejs lib and prompt “generate 3D world and make no mistake” are new orange/black.

Re: Karpathy’s Pelican

#197

Post by hansmayer »

andrewstuart wrote: post #147

This is equally bad as a pelican test. LLMs should be tested in the same way people should be tested for a job interview (but often aren’t) - with tasks RELEVANT to usage. So you don’t just randomly pick some random thing to make the LLM randomly do (like many job interviewers do). You start with clear statements about real world usage scenarios. THEN you come up with tests that give insight to how well the LLM/hob s…

[dead]

Re: Karpathy’s Pelican

#198

Post by fasterik »

barrenko wrote: post #191
fasterik wrote: post #168

Earlier quoted context omitted.

Agreed that software isn't going to fix our institutions. But we need 0 new software? AlphaFold won its creators a Nobel prize in chemistry and solved a research problem that feeds into every area of biology, medical research, and drug discovery. We still have an untold number of unsolved problems related to human health, food production, energy production, infrastructure, transportation, education... The list is pra…

Agree for stuff like alphafold, for the likes of education, imho, it's been solved for at least a couple of centuries. Textbook and blackboard > ipad.

For childhood education sure, we don't really need high tech solutions there. But what about training the next generation of mathematicians, physicists, and engineers? Even decades ago, computer algebra systems and numerical solvers started to become indispensable, at least in many subfields. Now we're moving into the territory of automated proofs of mathematical conjectures. The state of the art is going to keep improving, and education is going to have to adapt to keep pace.

Re: Karpathy’s Pelican

#199

Post by HarHarVeryFunny »

lowbloodsugar wrote: post #151
HarHarVeryFunny wrote: post #47

It seems pretty clear that Anthropic models have been specifically trained to be good at generating three.js (JavaScript 3-D Graphics) code, so given current state of AI code generation in general, I don't find three.js models/animations as indicative of anything other than the model's ability to write three.js code. When Fable was first released the day-1 demos of it on Twitter (presumably from people who were given…

Don’t worry. I’m sure they’re not training it to be good at things like writing database backends, financial services, logistics systems, user interfaces, or anything of economic value. As long as you’re not working on three.js specifically, I’m sure Anthropic isn’t making any progress you should be worried about.

Do you feel that rendering a 3-D picture of Bilbo Baggins' house is a good indicator of generating economic value (i.e "AGI" in OpenAI parlance)?

Re: Karpathy’s Pelican

#200

Post by swe_dima »

In my experience SVGs are still too hard for LLMs.

I gave Fable a jpeg and asked to draw an SVG, using a loop that renders the SVG into an image so Fable can inspect it.

Results looked like drawing of a 5 year old.

Post reply on HN