Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT Unicorn has drawn a unicorn

gpt-unicorn.adamkdean.co.uk

Post reply on HN
Print view View original

61–70 of 207 posts

Re: GPT Unicorn has drawn a unicorn

#61

Post by bhouston »

vasco wrote: post #57
bhouston wrote: post #48

Earlier quoted context omitted.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

If you look at the examples from April, only 2 or 3 can be counted as unicorns. If in 3 months it's the reverse and only 2 or 3 can't be counted as unicorns, that would show a progressive improvement in the model. I agree we shouldn't take much from day N-1 to day N as there will be a lot of variance, but this can show us progressive improvement over model updates. Perhaps in a few months half of the generated pictur…

I do agree with your comment, especially this part:

> We just need to wait longer for the signal to break through the noise.

Currently, what we are observing is primarily noise with very little signal.

Re: GPT Unicorn has drawn a unicorn

#62

Post by fsloth »

bhouston wrote: post #48
calderknight wrote: post #38

Earlier quoted context omitted.

I don't see the bad logic.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

I don't think anyone claims this is an iterative linear measure, rather than a step function.

SVG can present arbitrarily complex graphics. The underlying display tech supports what ever fidelity GPT will eventually mature into.

Has GPT plateaud? Will it be stuck forever at this hilariously naive level of competence at SVG art? Will it mature into Midjourney level competence? I have no frigging clue. Since the token context is so small I imagine it will put limitations to the complexity of SVG art piece.

But I don't know. And it's fun to have a daily measure.

As a software engineer with a penchant for graphics asking GPT to draw complex graphic shapes was one of the first tests I did for it. It's extremely interesting for me to collect progress data, no matter how noisy.

I have no idea if GPT will ever mature beyond these squiggles but if it does, this track record will have at least considerable artistic value, if nothing else.

Re: GPT Unicorn has drawn a unicorn

#63

Post by worldsayshi »

sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

Even if a single version of gpt4 would be deterministic any change done to the model would probably introduce enough noise to make it impossible to make any conclusions on a few samples?

Re: GPT Unicorn has drawn a unicorn

#64

Post by bhouston »

calderknight wrote: post #51
bhouston wrote: post #48

Earlier quoted context omitted.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

It takes a picture from GPT every day, so we'll be able to see if this was a fluke by looking at future days' outputs. I think it will work to track changes in the model.

Yes, this makes sense. You are agreeing with me. In order to see if it is just a fluke or not, you need a lot of samples so you can characterize the distribution yourself and try to see if it changed.

Re: GPT Unicorn has drawn a unicorn

#65

Post by throwuwu »

sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

If the model understood the spacial relationships as well as the one that produced the original drawings of a unicorn then variance in the choice of the next token should produce many similar but somewhat different images of unicorns. None of the images until today bear any resemblance to the original images.

Re: GPT Unicorn has drawn a unicorn

#66

Post by [deleted] »

sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

[deleted]

Re: GPT Unicorn has drawn a unicorn

#67

Post by quickthrower2 »

xnorswap wrote: post #32
dannyw wrote: post #27

Earlier quoted context omitted.

This part: https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib...

That'll do it! I wonder how many retries it averages, the images list "tokens", but it's a bit cryptic to work out how to translate that to attempts?

You can also get the token probs and only sample from the ones that would be valid for each token.

Re: GPT Unicorn has drawn a unicorn

#68

Post by rcme »

sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

I'm not sure what you mean by random sampling. If I sample a random SVG, I wouldn't expect to look like anything, let alone roughly like a unicorn.

Re: GPT Unicorn has drawn a unicorn

#69

Post by hk__2 »

kachnuv_ocasek wrote: post #24
suby wrote: post #8

I'm confused as to why this would see any improvement over time. Looking at the code, it's by default hitting the gpt 3.5-turbo API. Maybe I'm misremembering, but I thought I've seen statements from people working at OpenAI where it's been claimed that the API is static, we'd be informed of any changes to the underlying model. Is the model actually receiving updates? edit: Looking at previous days, too, it doesn't ex…

It's using GPT-4, see the config file: https://github.com/adamkdean/gpt-unicorn/blob/357a466c366f67...

It’s using GPT-4 by default, but we can’t know what it uses for real since that’s in the environment config.
Post reply on HN