Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs, illustrated by pelicans on bicycles

simonwillison.net

Post reply on HN
Print view View original

81–90 of 244 posts

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#81

Post by adrian17 »

> This was one of the most successful product launches of all time. They signed up 100 million new user accounts in a week! They had a single hour where they signed up a million new accounts, as this thing kept on going viral again and again and again.

Awkwardly, I never heard of it until now. I was aware that at some point they added ability to generate images to the app, but I never realized it was a major thing (plus I already had an offline stable diffusion app on my phone, so it felt less of an upgrade to me personally). With so much AI news each week, feels like unless you're really invested in the space, it's almost impossible to not accidentally miss or dismiss some big release.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#82

Post by aspenmayer »

loloquwowndueo wrote: post #60
cyanydeez wrote: post #40

Earlier quoted context omitted.

Humans absolutely do not work discretely.

They probably meant deterministically as opposed to probabilistically. Which also humans dont work like that :)

I thought they meant discreetly.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#85

Post by 6LLvveMx2koXfwn »

demosthanos wrote: post #49
simonw wrote: post #29

Earlier quoted context omitted.

It might not be 100% clear from the writing but this benchmark is mainly intended as a joke - I built a talk around it because it's a great way to make the last six months of model releases a lot more entertaining. I've been considering an expanded version of this where each model outputs ten images, then a vision model helps pick the "best" of those to represent that model in a further competition with other models.…

I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up. Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...

I would definitely say he had no intention of doing that and was doubling down on the original joke.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#86

Post by bredren »

Great writeup.

This measure of LLM capability could be extended by taking it into the 3D domain.

That is, having the model write Python code for Blender, then running blender in headless mode behind an API.

The talk hints at this but one shot prompting likely won’t be a broad enough measurement of capability by this time next year. (Or perhaps now, even)

So the test could also include an agentic portion that includes consultation of the latest blender documentation or even use of a search engine for blog entries detailing syntax and technique.

For multimodal input processing, it could take into account a particular photo of a pelican as the test subject.

For usability, the objects can be converted to iOS’s native 3d format that can be viewed in mobile safari.

I built this workflow, including a service for blender as an initial test of what was possible in October of 2022. It took post processing for common syntax errors back then but id imagine the newer LLMs would make those mistakes less often now.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#87

Post by azinman2 »

adrian17 wrote: post #81

> This was one of the most successful product launches of all time. They signed up 100 million new user accounts in a week! They had a single hour where they signed up a million new accounts, as this thing kept on going viral again and again and again. Awkwardly, I never heard of it until now. I was aware that at some point they added ability to generate images to the app, but I never realized it was a major thing (p…

Except this went very mainstream. Lots of turn myself into a muppet, what is the human equivalent for my dog, etc. TikTok is all over this.

It really is incredible.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#89

Post by ramesh31 »

landgenoot wrote: post #72

If you would give a human the SVG documentation and ask to write an SVG, I think the results would be quite similar.

>If you would give a human the SVG documentation and ask to write an SVG, I think the results would be quite similar.

It certainly would, and it would cost at minimum an hour of the human programmer's time at $50+/hr. Claude does it in seconds for pennies.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#90

Post by jodrellblank »

ben_w wrote: post #21
puttycat wrote: post #16

Earlier quoted context omitted.

You are right, but the companies making these models invest a lot of effort in marketing them as anything but probabilistic, i.e. making people think that these models work discretely like humans. In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. In any case, even if a model is probabilistic, if it had c…

> In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. Look upon these works, ye mighty, and despair: https://www.gianlucagimini.it/portfolio-item/velocipedia/

You claim those are drawn by people with "perfect knowledge about bikes" and "perfect drawing skills"?
Post reply on HN