Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs, illustrated by pelicans on bicycles

simonwillison.net

Post reply on HN
Print view View original

161–170 of 244 posts

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#161

Post by zurichisstained »

Wow, I love this benchmark - I've been doing something similar (as a joke for and much less frequently), where I ask multiple models to attempt to create a data structure like:

``` const melody = [ { freq: 261.63, duration: 'quarter' }, // C4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 293.66, duration: 'triplet' }, // D4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 329.63, duration: 'half' }, // E4 ] ```

But with the intro to Smoke on the Water by Deep Purple. Then I run it through the Web Audio API and see how it sounds.

It's never quite gotten it right, but it's gotten better, to the point where I can ask it to make a website that can play it.

I think yours is a lot more thoughtful about testing novelty, but its interesting to see them attempt to do things that they aren't really built for (in theory!).

https://codepen.io/mvattuone/pen/qEdPaoW - ChatGPT 4 Turbo

https://codepen.io/mvattuone/pen/ogXGzdg - Claude Sonnet 3.7

https://codepen.io/mvattuone/pen/ZYGXpom - Gemini 2.5 Pro

Gemini is by far the best sounding one, but it's still off. I'd be curious how the latest and greatest (paid) versions fare.

(And just for comparison, here's the first time I did it... you can tell I did the front-end because there isn't much to it!) https://nitter.space/mvattuone/status/1646610228748730368#m

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#162

Post by vunderba »

neepi wrote: post #2

My only take home is they are all terrible and I should hire a professional.

This test isn't really about the quality of the image itself (multimodals like gpt-image-1 or even standard diffusion models would be far superior) - it's about following a spec that describes how to draw.

A similar test would be if you asked for the pelican on a bicycle through a series of LOGO instructions.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#163

Post by famouswaffles »

simonw wrote: post #51
fzzzy wrote: post #50

Earlier quoted context omitted.

Even if it is a joke, having a consistent methodology is useful. I did it for about a year with my own private benchmark of reasoning type questions that I always applied to each new open model that came out. Run it once and you get a random sample of performance. Got unlucky, or got lucky? So what. That's the experimental protocol. Running things a bunch of times and cherry picking the best ones adds human bias, and…

It wasn't until I put these slides together that I realized quite how well my joke benchmark correlates with actual model performance - the "better" models genuinely do appear to draw better pelicans and I don't really understand why!

LLMs also have a 'g factor' https://www.sciencedirect.com/science/article/pii/S016028962...

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#165

Post by micromacrofoot »

Jedd wrote: post #96
thierrydamiba wrote: post #94

Earlier quoted context omitted.

The big trend was around the ghiblification of images. Those images were everywhere for a period of time.

Yeah, but so were the bored ape NFTs - none of these ephemeral fads are any indication of quality, longevity, legitimacy, or interest.

they're not but I'm already seeing ai generated images on billboards for local businesses, they're in production workflows now and they aren't going anywhere

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#166

Post by djhn »

baq wrote: post #151
senthil_rajasek wrote: post #144

Earlier quoted context omitted.

"My wife is using ChatGPT to change photos (still is to this day), she didn’t use it or any other LLM until that feature hit." This is deja vu, except instead of ChatGPT to edit photos it was instagram a decade ago.

You either haven’t tried it or are just trolling.

I tried it and I don’t get it. What and where are the legal usecases? What can you do with these low-resolution images?

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#167

Post by adrian17 »

bufferoverflow wrote: post #152
adrian17 wrote: post #81

> This was one of the most successful product launches of all time. They signed up 100 million new user accounts in a week! They had a single hour where they signed up a million new accounts, as this thing kept on going viral again and again and again. Awkwardly, I never heard of it until now. I was aware that at some point they added ability to generate images to the app, but I never realized it was a major thing (p…

Have you missed how everyone was Ghiblifying everything?

I saw that, I just didn't connect it with newly added multimodal image generation. I knew variations of style transfer (or LoRA for SD) were possible for years, so I assumed it exploded in popularity purely as a meme, not due to OpenAI making it much more accessible.

Again, I was aware that they added image generation, just not how much of a deal it turned out to be. Think of it like me occasionally noticing merchandise and TV trailers for a new movie without realizing it became the new worldwide box office #1.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#168

Post by senthil_rajasek »

baq wrote: post #151
senthil_rajasek wrote: post #144

Earlier quoted context omitted.

"My wife is using ChatGPT to change photos (still is to this day), she didn’t use it or any other LLM until that feature hit." This is deja vu, except instead of ChatGPT to edit photos it was instagram a decade ago.

You either haven’t tried it or are just trolling.

I am contrasting how instagram filters gave users some control and increased user base and how today editing photos with LLMs is doing the same and pulling in a wider user base.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#169

Post by MattRix »

adrian17 wrote: post #81

> This was one of the most successful product launches of all time. They signed up 100 million new user accounts in a week! They had a single hour where they signed up a million new accounts, as this thing kept on going viral again and again and again. Awkwardly, I never heard of it until now. I was aware that at some point they added ability to generate images to the app, but I never realized it was a major thing (p…

To be clear: they already had image generation in ChatGPT, but this was a MUCH better one than what they had previously. Even for you with your stable diffusion app, it would be a significant upgrade. Not just because of image quality, but because it can actually generate coherent images and follow instructions.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#170

Post by MattRix »

isx726552 wrote: post #149

> I’ve been feeling pretty good about my benchmark! It should stay useful for a long time... provided none of the big AI labs catch on. > And then I saw this in the Google I/O keynote a few weeks ago, in a blink and you’ll miss it moment! There’s a pelican riding a bicycle! They’re on to me. I’m going to have to switch to something else. Yeah this touches on an issue that makes it very difficult to have a discussion…

This is why things like the ARC Prize are better ways of approaching this: https://arcprize.org
Post reply on HN