Why would the model change over time when asking the same question? Just it's generation dataset for generating similar images? Or is this just tracking GPT's explicit model improvements over time?
GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
11–20 of 109 posts
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#12> The idea behind GPT Unicorn is quite simple: every day, GPT-4 will be asked to draw a unicorn in SVG format. This daily interaction with the model will allow us to observe changes in the model over time, as reflected in the output. Is it useful to do this every day? Correct me if I'm wrong, but my understanding is that OpenAI does not update the models available in production incrementally on a day-to-day basis.
Similarly the quality of the model can't be judged with a single sample. These end up canceling out.
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#13Earlier quoted context omitted.
It's not just image generation the rlhf worsens too. Calibration (confidence on solving a question in relation to ability to solve that problem) went from excellent to non existent. and you can see from the report that the base model performed better on a number of tests. Basically a dumber model.
Are you saying this specifically for the GPT-4 API endpoint compared to idealized described GPT-4 from the paper?
yes the public api (or on paid chatGPT) vs the base model from the paper
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#14Would be useful if the prompts used to generate the drawing code were included in the site.
They appear to be here: https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib... { role: 'system', content: `You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text.` }, { role: 'user', content: `Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks.` }
Sadly it outputs raw svg code so you have to save it locally as .svg to see it. Or just insert it into an HTML page via devtools if you're lazy like me.
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#15As is noted in the paper from which this is inspired from: GPT-4's image generation capabilities were severely diminished by the instruction/safety-tuning process. Unfortunately this means the currently available model from the API won't be very capable - certainly not as capable as the early version of GPT-4 that Microsoft had access to. edit: I'm specifically referring to the "image generation by trickery (e.g. SVG…
It's not just image generation the rlhf worsens too. Calibration (confidence on solving a question in relation to ability to solve that problem) went from excellent to non existent. and you can see from the report that the base model performed better on a number of tests. Basically a dumber model.
Not dumber. More biased.
Important distinction, especially if we're looking to push back out towards the Pareto Frontier of the problem.
RLHF is still very much in its infancy and does not maximize the bias-variance tradeoff by a long shot, in my personal experience.
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#16Why would the model change over time when asking the same question? Just it's generation dataset for generating similar images? Or is this just tracking GPT's explicit model improvements over time?
The models seem to have been changing in the background, though as another commenter pointed out.... having a variance-calibrayion baseline for humans would be great too. :'))))
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#17Earlier quoted context omitted.
It's not just image generation the rlhf worsens too. Calibration (confidence on solving a question in relation to ability to solve that problem) went from excellent to non existent. and you can see from the report that the base model performed better on a number of tests. Basically a dumber model.
Not dumber. More biased. Important distinction, especially if we're looking to push back out towards the Pareto Frontier of the problem. RLHF is still very much in its infancy and does not maximize the bias-variance tradeoff by a long shot, in my personal experience.
No dumber. Sure more biased too if you want but also dumber. Open ai have indicated as much.
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#18Earlier quoted context omitted.
It's not just image generation the rlhf worsens too. Calibration (confidence on solving a question in relation to ability to solve that problem) went from excellent to non existent. and you can see from the report that the base model performed better on a number of tests. Basically a dumber model.
Not dumber. More biased. Important distinction, especially if we're looking to push back out towards the Pareto Frontier of the problem. RLHF is still very much in its infancy and does not maximize the bias-variance tradeoff by a long shot, in my personal experience.
My understanding is that OpenAI did indeed find diminished capability across a range of tasks after doing RLHF. You're correct to question this though - as I believe the opposite was true of GPT-3 where it improved certain tasks.
The benefits from a business perspective were still clear however, and of course the instruction-tuned GPT-4 model still outperformed GPT-3, in general.
There are probably some weird edge cases and nuances that I'm missing - and I'd be happy to be corrected.
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#19Perhaps you should ask it to draw you a sheep.
Re: GPT Unicorn: A Daily Exploration of GPT-4's Image Generation Capabilities
#20Would be useful if the prompts used to generate the drawing code were included in the site.
They appear to be here: https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib... { role: 'system', content: `You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text.` }, { role: 'user', content: `Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks.` }
“You are a helpful assistant” seems like it is always included in these sort of prompts. I wonder if it really helps…