GPT Unicorn has drawn a unicorn • Hacker News Board

Re: GPT Unicorn has drawn a unicorn

#51

Post by calderknight » Thu, Aug 10, 2023, 10:42 AM UTC

bhouston wrote: post #48

calderknight wrote: post #38

Earlier quoted context omitted.

I don't see the bad logic.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

It takes a picture from GPT every day, so we'll be able to see if this was a fluke by looking at future days' outputs. I think it will work to track changes in the model.

Re: GPT Unicorn has drawn a unicorn

#52

Post by MauranKilom » Thu, Aug 10, 2023, 10:42 AM UTC

sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

Yes, nothing about GPT4 changed today. But that's not the goal of the project (although I can't speak for the intentions of the submitter here).

Currently there are two different GPT4 models represented in the samples, with quite significant quality difference between them. The quality (and variance in quality within a single model!) is interesting to see in such a comparison.

Re: GPT Unicorn has drawn a unicorn

#53

Post by horse_dung » Thu, Aug 10, 2023, 10:43 AM UTC

devjab wrote: post #16

Is it just me or doesn't image-2023-04-25 look like a unicorn as well?

4-22 is my favourite… hahaha

Re: GPT Unicorn has drawn a unicorn

#54

Post by pietroppeter » Thu, Aug 10, 2023, 10:44 AM UTC

Frummy wrote: post #40

pietroppeter wrote: post #31

Earlier quoted context omitted.

Is there some reference or explanation on why the model it is non deterministic at temperature 0?

https://news.ycombinator.com/item?id=37006224

Thanks!

Re: GPT Unicorn has drawn a unicorn

#55

Post by MauranKilom » Thu, Aug 10, 2023, 10:44 AM UTC

bhouston wrote: post #48

calderknight wrote: post #38

Earlier quoted context omitted.

I don't see the bad logic.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

If you look closely, you can see that the model version is attached to each image, and the original blog post clearly states that asking the same model over multiple days is how they intend to track the variance. No bad logic there imo.

Re: GPT Unicorn has drawn a unicorn

#56

Post by fastball » Thu, Aug 10, 2023, 10:44 AM UTC

gpjanik wrote: post #45

The prompt must be particularly bad. I managed to get a nicely looking unicorn at the first (and every subsequent) attempt.

The prompts are thus:

> system: You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text.

> user: Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks.

What were yours?

Re: GPT Unicorn has drawn a unicorn

#57

Post by vasco » Thu, Aug 10, 2023, 10:46 AM UTC

bhouston wrote: post #48

calderknight wrote: post #38

Earlier quoted context omitted.

I don't see the bad logic.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

If you look at the examples from April, only 2 or 3 can be counted as unicorns. If in 3 months it's the reverse and only 2 or 3 can't be counted as unicorns, that would show a progressive improvement in the model. I agree we shouldn't take much from day N-1 to day N as there will be a lot of variance, but this can show us progressive improvement over model updates.

Perhaps in a few months half of the generated pictures will look like unicorns, perhaps in more months they will all be unicorns but 2 or 3 will look way more detailed instead of drawn by a 4 year old, etc. We just need to wait longer for the signal to break through the noise.

Re: GPT Unicorn has drawn a unicorn

#58

Post by andai » Thu, Aug 10, 2023, 10:46 AM UTC

tellarin wrote: post #9

suby wrote: post #8

I'm confused as to why this would see any improvement over time. Looking at the code, it's by default hitting the gpt 3.5-turbo API. Maybe I'm misremembering, but I thought I've seen statements from people working at OpenAI where it's been claimed that the API is static, we'd be informed of any changes to the underlying model. Is the model actually receiving updates? edit: Looking at previous days, too, it doesn't ex…

Yes, the models are updated officially around every three months, with a notice you can still use the previous version for a time until it is decommissioned. Some people claim there are also unannounced changes, but I can't vouch for that. The daily variation is likely due to temperature. To make the response less repetitive.

Wasn't there a study recently that tracked the performance of GPT over time and found significant drop in quality? Did those drops occur at official model changes, or at other times? (i.e. unannounced changes for safety or cost reduction)

I mean, if I was OpenAI, I probably wouldn't make an announcement like "we've just quantized the model and increased our profit margins significantly! The only change on your end will be a slightly dumber model. (Don't worry! Most users won't even notice!)"

Re: GPT Unicorn has drawn a unicorn

#59

Post by Teever » Thu, Aug 10, 2023, 10:53 AM UTC

bhouston wrote: post #48

calderknight wrote: post #38

Earlier quoted context omitted.

I don't see the bad logic.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

you're looking at it from the wrong time scale.

The difference over months or years is what is interesting.

Re: GPT Unicorn has drawn a unicorn

#60

Post by pbw » Thu, Aug 10, 2023, 10:55 AM UTC

I made some drawings with GPT-4 Code Interpreter and Pillow. I think with its sub-mage and image composition features you could make some detailed drawings if you were clever about it.

[1] https://metastable.org/draw.html