Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs, illustrated by pelicans on bicycles

simonwillison.net

Post reply on HN
Print view View original

181–190 of 244 posts

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#181

Post by gscott »

dirtyhippiefree wrote: post #23

Here’s the spot where we see who’s TL;DR… > Claude 4 will rat you out to the feds! >If you expose it to evidence of malfeasance in your company, and you tell it it should act ethically, and you give it the ability to send email, it’ll rat you out.

I am interested in this ratting you out thing. At some point you have a video feed into AI from a Jarvis like headset device, you walking down the street and cross the street in the middle not at a sidewalk... does it rat you out? Does it make a list of every crime no matter how small? Or just the big ones?

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#182

Post by andrepd »

bufferoverflow wrote: post #152
adrian17 wrote: post #81

> This was one of the most successful product launches of all time. They signed up 100 million new user accounts in a week! They had a single hour where they signed up a million new accounts, as this thing kept on going viral again and again and again. Awkwardly, I never heard of it until now. I was aware that at some point they added ability to generate images to the app, but I never realized it was a major thing (p…

Have you missed how everyone was Ghiblifying everything?

Oh you mean the trend of the day on the social media monoculture? I don't take that as an indicator of any significance.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#185

Post by dgfitz »

derwiki wrote: post #158
haiku2077 wrote: post #111

Earlier quoted context omitted.

Congratulations, you are almost fully unplugged from social media. This product launch was a huge mainstream event; for a few days GPT generated images completely dominated mainstream social media.

Not sure if this is sarcasm or sincere, but I will take it as sincere haha. I came back to work from parental leave and everyone had that same Studio Ghiblized image as their Slack photo, and I had no idea why. It turns out you really can unplug from social media and not miss anything of value: if it’s a big enough deal you will find out from another channel.

I missed it until this thread. I think I’m proud of myself.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#187

Post by ojosilva »

zurichisstained wrote: post #161

Wow, I love this benchmark - I've been doing something similar (as a joke for and much less frequently), where I ask multiple models to attempt to create a data structure like: ``` const melody = [ { freq: 261.63, duration: 'quarter' }, // C4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 293.66, duration: 'triplet' }, // D4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 329.63, duration: 'half'…

Drawbacks for using a pelican on a bicycle svg: it's a very open-ended prompt, no specific criteria to judge, and lately the svg all start to look similar, or at least like they accomplished the same non-goals (there's a pelican, there's a bicycle and I'm not sure its feet should be on the saddle or on the pedals), so it's hard to agree on which is better. And, certainly, having a LLM as a judge, the entire game becomes double-hinged and who knows what to think.

Also, if it becomes popular, training sets may pick it up and improve models unfairly and unrealistically. But that's true of any known benchmark.

Side note: I'd really like to see the Language Benchmark Game become a prompt based languages * models benchmark game. So we could say model X excels at Python Fasta, etc. although then the risk is that, again, it becomes training set and the whole thing self-rigs itself.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#188

Post by zurichisstained »

dr_kretyn wrote: post #174
zurichisstained wrote: post #161

Wow, I love this benchmark - I've been doing something similar (as a joke for and much less frequently), where I ask multiple models to attempt to create a data structure like: ``` const melody = [ { freq: 261.63, duration: 'quarter' }, // C4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 293.66, duration: 'triplet' }, // D4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 329.63, duration: 'half'…

I'm slightly confused by your example. What's the actual prompt? Is your expectation that a text model is going to know how to perform the exact song in audio?

Ohhh absolutely not, that would be pretty wild - I just wanted to see if it could understand musical notation enough to come up with the correct melody.

I know there are far better ways to do gen AI with music, this was just a joke prompt that worked far better than I expected.

My naive guess is all of the guitar tabs and signal processing info it's trained on gives it the ability to do stuff like this (albeit not very well).

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#190

Post by telotortium »

demosthanos wrote: post #49
simonw wrote: post #29

Earlier quoted context omitted.

It might not be 100% clear from the writing but this benchmark is mainly intended as a joke - I built a talk around it because it's a great way to make the last six months of model releases a lot more entertaining. I've been considering an expanded version of this where each model outputs ten images, then a vision model helps pick the "best" of those to represent that model in a further competition with other models.…

I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up. Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...

Yeah, Simon needs to release a new benchmark under a pen name, like Stephen King did with Richard Bachman.
Post reply on HN