Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

adamkdean.co.uk

Post reply on HN
Print view View original

21–30 of 109 posts

Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

#21

Post by atleastoptimal »

if GPT-4 will update based on recent web training data, the fact that people are bringing much more attention to the "draw a unicorn" task magnifies the chance someone will have posted a perfect version of an svg unicorn, leading the model to leverage that rather than the aim of this experiment which I imagine is GPT-4's capacity to extrapolate.

EDIT: Also it makes no sense to constantly retry it every day on the gpt-4-0314 model, since OpenAI specified that that is a snapshot model that will not be updated.

Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

#22

Post by pps »

dmix wrote: post #11

Why would the model change over time when asking the same question? Just it's generation dataset for generating similar images? Or is this just tracking GPT's explicit model improvements over time?

"GPT 5 Will be Released 'Incrementally' - 5 Points from Brockman Statement" - https://www.youtube.com/watch?v=1NAmLp5i4Ps

Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

#23

Post by sacred_numbers »

dangond wrote: post #6

> The idea behind GPT Unicorn is quite simple: every day, GPT-4 will be asked to draw a unicorn in SVG format. This daily interaction with the model will allow us to observe changes in the model over time, as reflected in the output. Is it useful to do this every day? Correct me if I'm wrong, but my understanding is that OpenAI does not update the models available in production incrementally on a day-to-day basis.

They do update the model in the background, although I'm not sure how often or how much they update it. To avoid issues with this practice they offer gpt-4-0314 which says this in the documentation:

"Snapshot of gpt-4 from March 14th 2023. Unlike gpt-4, this model will not receive updates, and will only be supported for a three month period ending on June 14th 2023."

Unfortunately this experiment is using the frozen snapshot model gpt-4-0314 instead of the unfrozen gpt-4 or gpt-4-32k models, so any differences are literally 100% noise. This would be a somewhat interesting experiment if someone were to use an unfrozen model, though. I do appreciate the author for captioning the images with the exact model they used for generation so that this bug could be caught quickly.

[0]https://platform.openai.com/docs/models/gpt-4

Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

#25

Post by ratg13 »

sp0rk wrote: post #7

Did you generate a bunch all at once before starting to get some idea of what the natural variance looks like? I would think it's important to verify some level of progression over time, because with the current four it seems entirely possible that the examples could have all been generated at the same time with no changes to the model.

Aren’t they using the March 14 model like the general public?

It’s frozen in time, there are no updates to it..

All of these will be drawn using the same model until they push a new update, or you switch to a different GPT

But I already think they proved the point that the generation is random enough that it would be extremely difficult to track progress this way.

Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

#26

Post by LeoPanthera »

bee_rider wrote: post #20
abrichr wrote: post #4

Earlier quoted context omitted.

They appear to be here: https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib... { role: 'system', content: `You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text.` }, { role: 'user', content: `Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks.` }

“You are a helpful assistant” seems like it is always included in these sort of prompts. I wonder if it really helps…

It's quite funny to tell it that it is an unhelpful assistant. During the first few responses it is amusingly obstinate.

It always seems to revert back to "helpful assistant" after a few messages, whatever the prompt says.

Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

#28

Post by psychphysic »

famouswaffles wrote: post #17
tbalsam wrote: post #15

Earlier quoted context omitted.

Not dumber. More biased. Important distinction, especially if we're looking to push back out towards the Pareto Frontier of the problem. RLHF is still very much in its infancy and does not maximize the bias-variance tradeoff by a long shot, in my personal experience.

No dumber. Sure more biased too if you want but also dumber. Open ai have indicated as much.

Also generally less creative and insightful.

"No I won't do it" becomes a good option no matter what if you turn safety too high.

Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities

#30

Post by mustacheemperor »

Varqu wrote: post #27

Is anyone else also getting tired of seeing "GPT" prefix / suffix in the name of 90% new AI-related products?

Given this is a process specifically to evaluate the changing performance of GPT-4 over time, it seems appropriate.
Post reply on HN