Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs, illustrated by pelicans on bicycles

simonwillison.net

Post reply on HN
Print view View original

111–120 of 244 posts

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#111

Post by haiku2077 »

adrian17 wrote: post #81

> This was one of the most successful product launches of all time. They signed up 100 million new user accounts in a week! They had a single hour where they signed up a million new accounts, as this thing kept on going viral again and again and again. Awkwardly, I never heard of it until now. I was aware that at some point they added ability to generate images to the app, but I never realized it was a major thing (p…

Congratulations, you are almost fully unplugged from social media. This product launch was a huge mainstream event; for a few days GPT generated images completely dominated mainstream social media.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#112

Post by Retric »

ben_w wrote: post #101
jodrellblank wrote: post #90

Earlier quoted context omitted.

You claim those are drawn by people with "perfect knowledge about bikes" and "perfect drawing skills"?

More that "these models work … like humans" (discretely or otherwise) does not imply the quotation. Most humans do not have perfect drawing skills and perfect knowledge about bikes and birds, they do not output such a simple drawing correctly 100% of the time. "Average human" is a much lower bar than most people want to believe, mainly because most of us are average on most skills, and also overestimate our own compe…

It’s not that humans have perfect drawing skills, it’s that humans can judge their performance and get better over time.

Ask 100 random people to draw a bike and in 10 minutes and they’ll on average suck while still beating the LLM’s here. Give em an incentive and 10 months and the average person is going to be able to make at least one quite decent drawing of a bike.

The cost and speed advantage of LLM’s is real as long as you’re fine with extremely low quality. Ask a model for 10,000 drawings so you can pick the best and you get a marginal improvements based on random chance at a steep price.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#113

Post by pjs_ »

puttycat wrote: post #19
qwertytyyuu wrote: post #15

https://imgur.com/a/mzZ77xI here are a few i tried the models, looks like the newer vesion of gemini is another improvement?

The bicycle are still very far from actual ones.

https://www.gianlucagimini.it/portfolio-item/velocipedia/

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#114

Post by throwaway31131 »

demosthanos wrote: post #49
simonw wrote: post #29

Earlier quoted context omitted.

It might not be 100% clear from the writing but this benchmark is mainly intended as a joke - I built a talk around it because it's a great way to make the last six months of model releases a lot more entertaining. I've been considering an expanded version of this where each model outputs ten images, then a vision model helps pick the "best" of those to represent that model in a further competition with other models.…

I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up. Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...

I’d say it doesn’t really matter. There is no universally good benchmark and really they should only be used to answer very specific questions which may or may not be relevant to you.

Also, as the old saying goes, the only thing worse than using benchmarks is not using benchmarks.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#115

Post by baq »

Jedd wrote: post #96
thierrydamiba wrote: post #94

Earlier quoted context omitted.

The big trend was around the ghiblification of images. Those images were everywhere for a period of time.

Yeah, but so were the bored ape NFTs - none of these ephemeral fads are any indication of quality, longevity, legitimacy, or interest.

It’s hard to think of a worse analogy TBH. My wife is using ChatGPT to change photos (still is to this day), she didn’t use it or any other LLM until that feature hit. It is a fad, but it’s also a very useful tool.

Ape NFTs are… ape NFTs. Useless. Pointless. Negative value for most people.

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#116

Post by mormegil »

diggan wrote: post #73
landgenoot wrote: post #72

If you would give a human the SVG documentation and ask to write an SVG, I think the results would be quite similar.

Lets give it a try, if you're willing to be the experiment subject :) The prompt is "Generate an SVG of a pelican riding a bicycle" and you're supposed to write it by hand, so no graphical editor. The specification is here: https://www.w3.org/TR/SVG2/ I'm fairly certain I'd lose interest in getting it right before I got something better than most of those.

Did the testing prompt for LLMs include a clause forbidding the use of any tools? If not, why are you adding it here?

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#117

Post by Ygg2 »

nine_k wrote: post #108

Am I the only one who can't but see these attempts much like attempts of a kid learning to draw?

Yes. Kids don't draw that good of a line at the start.

Here is better example of start https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTfTfAA...

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#119

Post by diggan »

mormegil wrote: post #116
diggan wrote: post #73

Earlier quoted context omitted.

Lets give it a try, if you're willing to be the experiment subject :) The prompt is "Generate an SVG of a pelican riding a bicycle" and you're supposed to write it by hand, so no graphical editor. The specification is here: https://www.w3.org/TR/SVG2/ I'm fairly certain I'd lose interest in getting it right before I got something better than most of those.

Did the testing prompt for LLMs include a clause forbidding the use of any tools? If not, why are you adding it here?

The models that are being put under the "Pelican" testing don't use a GUI to create SVGs (either via "tools" or anything else), they're all Text Generation models so they exclusively use text for creating the graphics.

There are 31 posts listed under "pelican-riding-a-bicycle" in case you wanna inspect the methodology even closer: https://simonwillison.net/tags/pelican-riding-a-bicycle/

Re: The last six months in LLMs, illustrated by pelicans on bicycles

#120

Post by simonw »

mormegil wrote: post #116
diggan wrote: post #73

Earlier quoted context omitted.

Lets give it a try, if you're willing to be the experiment subject :) The prompt is "Generate an SVG of a pelican riding a bicycle" and you're supposed to write it by hand, so no graphical editor. The specification is here: https://www.w3.org/TR/SVG2/ I'm fairly certain I'd lose interest in getting it right before I got something better than most of those.

Did the testing prompt for LLMs include a clause forbidding the use of any tools? If not, why are you adding it here?

The way I run the pelican on a bicycle benchmark is to use this exact prompt:

  Generate an SVG of a pelican riding a bicycle
And execute it via the model's API with all default settings, not via their user-facing interface.

Currently none of the model APIs enable tools unless you ask them to, so this method excludes the use of additional tools.

Post reply on HN