Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs, illustrated by pelicans on bicycles

simonwillison.net

Post reply on HN
Print view View original

11–20 of 244 posts

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#11

Post by namibj »

neepi wrote: post #9
dist-epoch wrote: post #4

Earlier quoted context omitted.

Most of them are text-only models. Like asking a person born blind to draw a pelican, based on what they heard it looks like.

That seems to be a completely inappropriate use case? I would not hire a blind artist or a deaf musician.

It's a proxy for abstract designing, like writing software or designing in a parametric CAD.

Most the non-math design work of applied engineering AFAIK falls under the umbrella that's tested with the pelican riding the bicycle. You have to make a mental model and then turn it into applicable instructions.

Program code/SVG markup/parametric CAD instructions don't really differ in that aspect.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#12

Post by __alexs »

neepi wrote: post #9
dist-epoch wrote: post #4

Earlier quoted context omitted.

Most of them are text-only models. Like asking a person born blind to draw a pelican, based on what they heard it looks like.

That seems to be a completely inappropriate use case? I would not hire a blind artist or a deaf musician.

I guess the idea is that by asking the model to do something that is inherently hard for it we might learn something about the baseline smartness of each model which could be considered a predictor for performance at other tasks too.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#13

Post by keiferski »

neepi wrote: post #2

My only take home is they are all terrible and I should hire a professional.

As the other guy said, these are text models. If you want to make images use something like Midjourney.

Promoting a pelican riding a bicycle makes a decent image there.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#14

Post by dmd »

neepi wrote: post #9
dist-epoch wrote: post #4

Earlier quoted context omitted.

Most of them are text-only models. Like asking a person born blind to draw a pelican, based on what they heard it looks like.

That seems to be a completely inappropriate use case? I would not hire a blind artist or a deaf musician.

Sorry, Beethoven, you just don’t seem to be a match for our org. Best of luck on your search!

You too, Monet. Scram.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#16

Post by puttycat »

nathan_phoenix wrote: post #6

My biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample. You wouldn't compare different random number generators by taking one sample from each and then concluding that generator 5 generates the highest numbers... Would be nicer to run the comparison with 10 images (or more) for each LLM and then average.

You are right, but the companies making these models invest a lot of effort in marketing them as anything but probabilistic, i.e. making people think that these models work discretely like humans.

In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time.

In any case, even if a model is probabilistic, if it had correctly learned the relevant knowledge you'd expect the output to be perfect because it would serve to lower the model's loss. These outputs clearly indicate flawed knowledge.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#17

Post by planb »

nathan_phoenix wrote: post #6

My biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample. You wouldn't compare different random number generators by taking one sample from each and then concluding that generator 5 generates the highest numbers... Would be nicer to run the comparison with 10 images (or more) for each LLM and then average.

And by a sample that has become increasingly known as a benchmark. Newer training data will contain more articles like this one, which naturally improves the capabilities of an LLM to estimate what’s considered a good „pelican on a bike“.
Post reply on HN