Coincidentally, I saw this headline after waking from a dream where a unicorn bashed through the window of my childhood kitchen, and I had to fight it off with a water gun.
Melatonin, not even once.
GPT Unicorn has drawn a unicorn
131–140 of 207 posts
Re: GPT Unicorn has drawn a unicorn
#132Earlier quoted context omitted.
He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…
I don't think anyone claims this is an iterative linear measure, rather than a step function. SVG can present arbitrarily complex graphics. The underlying display tech supports what ever fidelity GPT will eventually mature into. Has GPT plateaud? Will it be stuck forever at this hilariously naive level of competence at SVG art? Will it mature into Midjourney level competence? I have no frigging clue. Since the token…
I mean, how many humans can draw art by writing out svg? If that's not in the training set, I don't even see how GPT-4 gets much better at this over time.
Re: GPT Unicorn has drawn a unicorn
#133Earlier quoted context omitted.
The prompts are thus: > system: You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text. > user: Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks. What were yours?
My prompt was: Imagine you have to draw a SVG of an object. As a model that does not have any idea about how things look, you have to draw "blindly" - as there's no visual feedback, the only feasible tactic is to first list things components each thing consists of (e.g. for a car wheels, windows, chassis, bumpers, lights, etc.) with as much accuracy as you can, establish some constraints (e.g. in a horse legs come ou…
This prompt actually works pretty well. I suspect that because you added a step requiring the model to list out what it needs to draw first, it has a much easier time creating a reasonable SVG. https://chat.openai.com/share/be9cea50-d00d-4b05-ab54-b64619...
Re: GPT Unicorn has drawn a unicorn
#134Earlier quoted context omitted.
Yes, the models are updated officially around every three months, with a notice you can still use the previous version for a time until it is decommissioned. Some people claim there are also unannounced changes, but I can't vouch for that. The daily variation is likely due to temperature. To make the response less repetitive.
Wasn't there a study recently that tracked the performance of GPT over time and found significant drop in quality? Did those drops occur at official model changes, or at other times? (i.e. unannounced changes for safety or cost reduction) I mean, if I was OpenAI, I probably wouldn't make an announcement like "we've just quantized the model and increased our profit margins significantly! The only change on your end wi…
This one [1]? That tracks two distinct versions (0613 vs. 0314).
Also, IMO, the tasks they evaluate aren't useful (I rarely want my LLM to tell me whether 17077 is a prime number), and there's room for cherrypicking/survivorship bias. My guess is that OpenAI did something between 0314 and 0613 that shifted focus away from maths to other subjects.
Re: GPT Unicorn has drawn a unicorn
#135It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…
100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.
I'm sorry it hurts your head. I'm happy to sponsor a packet of paracetamol or some water if that helps. Ultimately, this is fun, not science. I'm just happy that after all these attempts, it finally got to a unicorn.
Re: GPT Unicorn has drawn a unicorn
#136Earlier quoted context omitted.
> The original post hurts my head with its bad logic. Huh? What "original post"? This is an experiment, today the model drew something resembling a unicorn. Tomorrow we will see how the experiment goes again. I see no associated analysis, so what makes your "head hurt".
https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploratio... > The idea behind GPT Unicorn is quite simple: every day, GPT-4 will be asked to draw a unicorn in SVG format. This daily interaction with the model will allow us to observe changes in the model over time, as reflected in the output.
Sure, each day it will draw a unicorn. Each time the model changes, we'll have a new group of drawings. No they're not drawn at T=0 but even at T=0 GPT-4 is not deterministic. This isn't science -- this is just a bit of fun.
Re: GPT Unicorn has drawn a unicorn
#137Earlier quoted context omitted.
I agree that trying to determine the distribution of these drawings is hard because it isn't a simple floating point number in its current form. But maybe you could covert it to a linear monotonic measure? You could pass it to an image recognition model and see record the degree to which it thinks it is an: animal horse unicorn Basically if it fails to be a unicorn, see if it is a horse and if it fails to be a horse…
> You could pass it to an image recognition model and see record the degree to which it thinks it is... We don't care what another algorithm "thinks". We want to see if what it draws is humanly interpretable as a unicorn.
Then simply train the model to predict whether a human can interpret it as a unicorn.
Re: GPT Unicorn has drawn a unicorn
#138Earlier quoted context omitted.
> You could pass it to an image recognition model and see record the degree to which it thinks it is... We don't care what another algorithm "thinks". We want to see if what it draws is humanly interpretable as a unicorn.
Then you could use mechanical turk to have them rate each image to figure out how close it is to a Unicorn...
Mechanical turk 'workers' use ChatGPT
Re: GPT Unicorn has drawn a unicorn
#139Earlier quoted context omitted.
> You could pass it to an image recognition model and see record the degree to which it thinks it is... We don't care what another algorithm "thinks". We want to see if what it draws is humanly interpretable as a unicorn.
Then you could use mechanical turk to have them rate each image to figure out how close it is to a Unicorn...
We could but this is also a fun project, which is why when I checked it today I was surprised that what I saw was not a turd with eyes (2023-05-18) nor a strange sea creature (2023-07-08) but something which, for the first time I think, actually resembled a unicorn.
I appreciate all the comments around determinism, sampling, scientific method, but as I said when I posted this just after building, it really is just for fun and to see, over time, if the general mish mash of outputs become more refined without any changes to the prompt (which doesn't aid it through CoT/ToT or improving on previous attempts etc.)
Re: GPT Unicorn has drawn a unicorn
#140It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…
Agreed. But have you seen the original talk? I believe he's trying to find an unicorn similar in style to the one generated by the original researcher. It's so sad that openai has a far more capable model internally that it can't give open access to because of safety (or any other argument).
I suspect the model is fundamentally the same underneath, but that various tricks like quantization are being performed in the deployed model to improve inference speed/cost at the expense of output quality.