Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs, illustrated by pelicans on bicycles

simonwillison.net

Post reply on HN
Print view View original

171–180 of 244 posts

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#171

Post by simonw »

isx726552 wrote: post #149

> I’ve been feeling pretty good about my benchmark! It should stay useful for a long time... provided none of the big AI labs catch on. > And then I saw this in the Google I/O keynote a few weeks ago, in a blink and you’ll miss it moment! There’s a pelican riding a bicycle! They’re on to me. I’m going to have to switch to something else. Yeah this touches on an issue that makes it very difficult to have a discussion…

Honestly, if my stupid pelican riding a bicycle benchmark becomes influential enough that AI labs waste their time optimizing for it and produce really beautiful pelican illustrations I will consider that a huge personal win.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#172

Post by Aurornis »

mrkurt wrote: post #104
Jedd wrote: post #96

Earlier quoted context omitted.

Yeah, but so were the bored ape NFTs - none of these ephemeral fads are any indication of quality, longevity, legitimacy, or interest.

If we try really hard, I think we can make an exhaustive list of what viral fads on the internet are not. You made a small start. none of these ephemeral fads are any indication of quality, longevity, legitimacy, interest, substance, endurance, prestige, relevance, credibility, allure, staying-power, refinement, or depth.

100 million people didn’t sign up to make that one image meme and then never use it again.

That many signups is impressive no matter what. The attempts to downplay every aspect of LLM popularity are getting really tiresome.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#173

Post by Choco31415 »

isx726552 wrote: post #149

> I’ve been feeling pretty good about my benchmark! It should stay useful for a long time... provided none of the big AI labs catch on. > And then I saw this in the Google I/O keynote a few weeks ago, in a blink and you’ll miss it moment! There’s a pelican riding a bicycle! They’re on to me. I’m going to have to switch to something else. Yeah this touches on an issue that makes it very difficult to have a discussion…

Just tried that canard on GPT-4o and it failed:

"The word "strawberry" contains 2 letter r’s."

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#174

Post by dr_kretyn »

zurichisstained wrote: post #161

Wow, I love this benchmark - I've been doing something similar (as a joke for and much less frequently), where I ask multiple models to attempt to create a data structure like: ``` const melody = [ { freq: 261.63, duration: 'quarter' }, // C4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 293.66, duration: 'triplet' }, // D4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 329.63, duration: 'half'…

I'm slightly confused by your example. What's the actual prompt? Is your expectation that a text model is going to know how to perform the exact song in audio?

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#175

Post by jodrellblank »

Aurornis wrote: post #172
mrkurt wrote: post #104

Earlier quoted context omitted.

If we try really hard, I think we can make an exhaustive list of what viral fads on the internet are not. You made a small start. none of these ephemeral fads are any indication of quality, longevity, legitimacy, interest, substance, endurance, prestige, relevance, credibility, allure, staying-power, refinement, or depth.

100 million people didn’t sign up to make that one image meme and then never use it again. That many signups is impressive no matter what. The attempts to downplay every aspect of LLM popularity are getting really tiresome.

I think it sounds far more likely that 100M people signed up to poke at the latest viral novelty and create one meme, than that 100M people suddenly discovered they had a pressing long-term need for AI images all on the same day.

Doesn’t it?

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#176

Post by NohatCoder »

If you calculate ELO based on a round-robin tournament with all participants starting out on the same score, then the resulting ratings should simply correspond to the win count. I guess the algorithm in use take into account the order of the matches, but taking order into account is only meaningful when competitors are expected to develop significantly, otherwise it is just added noise, so we never want to do so in competitions between bots.

I also can't help but notice that the competition is exactly one match short, for some reason exactly one of the 561 possible pairings has not been included.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#177

Post by gretch »

jodrellblank wrote: post #175
Aurornis wrote: post #172

Earlier quoted context omitted.

100 million people didn’t sign up to make that one image meme and then never use it again. That many signups is impressive no matter what. The attempts to downplay every aspect of LLM popularity are getting really tiresome.

I think it sounds far more likely that 100M people signed up to poke at the latest viral novelty and create one meme, than that 100M people suddenly discovered they had a pressing long-term need for AI images all on the same day. Doesn’t it?

It's neither of these options in this false dichotomy.

100M people signed up and did at least 1 task. Then, most likely some % of them discovered it was a useful thing (if for nothing else than just to make more memes), and converted into a MAU.

If I had to use my intuition, I would say it's 5% - 10%, which represents a larger product launch than most developers will ever participate in, in the context of a single day.

Of course the ongoing stickiness of the MAU also depends on the ability of this particular tool to stay on top amongst increasing competition.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#178

Post by ben_w »

jodrellblank wrote: post #175
Aurornis wrote: post #172

Earlier quoted context omitted.

100 million people didn’t sign up to make that one image meme and then never use it again. That many signups is impressive no matter what. The attempts to downplay every aspect of LLM popularity are getting really tiresome.

I think it sounds far more likely that 100M people signed up to poke at the latest viral novelty and create one meme, than that 100M people suddenly discovered they had a pressing long-term need for AI images all on the same day. Doesn’t it?

While 100M signing up just for one pic is certainly possible, I note that several hundred million people regularly share photographs of their lunch, so it is very plausible that in signing up for the latest meme generator they found they liked the ability to generate custom images of whatever they consider to be pretty pictures every day.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#179

Post by simonw »

NohatCoder wrote: post #176

If you calculate ELO based on a round-robin tournament with all participants starting out on the same score, then the resulting ratings should simply correspond to the win count. I guess the algorithm in use take into account the order of the matches, but taking order into account is only meaningful when competitors are expected to develop significantly, otherwise it is just added noise, so we never want to do so in…

Yeah, that's a good call out: Elo isn't actually necessary if you can have every competitor battle every other competitor exactly once.

The missing match is because one single round was declared a draw by the model, and I didn't have time to run it again (the Elo stuff was very much rushed at the last minute.)

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#180

Post by threeseed »

joshuajooste05 wrote: post #135

Does anyone have any thoughts on privacy/safety regarding what he said about GPT memory. I had heard of prompt injection already. But, this seems different, completely out of humans control. Like even when you consider web search functionality, he is actually right, more and more, users are losing control over context. Is this dangerous atm? Do you think it will become more dangerous in the future when we chuck even…

I've had Cursor/Claude try to call rm -rf on my entire User directory before.

The issue is that LLMs have no ability to organise their memory by importance. Especially as the context size gets larger.

So when they are using tools they will become more dangerous over time.

Post reply on HN