Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT Unicorn has drawn a unicorn

gpt-unicorn.adamkdean.co.uk

Post reply on HN
Print view View original

41–50 of 207 posts

Re: GPT Unicorn has drawn a unicorn

#41

Post by calderknight »

tellarin wrote: post #9
suby wrote: post #8

I'm confused as to why this would see any improvement over time. Looking at the code, it's by default hitting the gpt 3.5-turbo API. Maybe I'm misremembering, but I thought I've seen statements from people working at OpenAI where it's been claimed that the API is static, we'd be informed of any changes to the underlying model. Is the model actually receiving updates? edit: Looking at previous days, too, it doesn't ex…

Yes, the models are updated officially around every three months, with a notice you can still use the previous version for a time until it is decommissioned. Some people claim there are also unannounced changes, but I can't vouch for that. The daily variation is likely due to temperature. To make the response less repetitive.

The site linked in the OP is interesting because it takes a picture from GPT every day, so we can see for ourselves if there is any difference with time. We can come back tomorrow and see what it has produced. If it has produces random squigly lines again, we might assume that today's success was just a fluke.

Re: GPT Unicorn has drawn a unicorn

#43

Post by CrzyLngPwd »

There's no progress from something that is definitely not a unicorn towards a unicorn, the images seem randomly bad.

I think it would be a lot of fun to give it the previous unicorn SVG attempt and ask it to make it more like a unicorn.

Re: GPT Unicorn has drawn a unicorn

#44

Post by luc4sdreyer »

pietroppeter wrote: post #31
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

Is there some reference or explanation on why the model it is non deterministic at temperature 0?

I'm not aware of anything concrete by OpenAI, but others have offered possible explanations.

One idea is that the cause is batched inference in sparse MoE (mixture of experts) models.

https://152334h.github.io/blog/non-determinism-in-gpt-4/

HN discussion: https://news.ycombinator.com/item?id=37006224

Re: GPT Unicorn has drawn a unicorn

#48

Post by bhouston »

calderknight wrote: post #38
bhouston wrote: post #21

Earlier quoted context omitted.

100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.

I don't see the bad logic.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model."

The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising.

If he wanted to track changes in the model for real, he would have to ask multiple questions per day and try to infer some type of distribution characterization and then see if that changes over time. That is much more complex and not what he is doing.

This is just a curious experiment that doesn't mean much.

Re: GPT Unicorn has drawn a unicorn

#49

Post by rav »

pietroppeter wrote: post #31
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

Is there some reference or explanation on why the model it is non deterministic at temperature 0?

One important source of non-determinism is from using massive parallelism together with floating point arithmetic. In real math, a sum of numbers has an exact value that doesn't change if you change which order the numbers are added up in, but floating point arithmetic addition is not associative in the same way as real math, and parallelism can cause numbers to be added in a different order from execution to execution, which is one cause of non-determinism.
Post reply on HN