Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT Unicorn has drawn a unicorn

gpt-unicorn.adamkdean.co.uk

Post reply on HN
Print view View original

81–90 of 207 posts

Re: GPT Unicorn has drawn a unicorn

#81

Post by nurettin »

bhouston wrote: post #21
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.

What if, instead of your ridiculous strawman, they believe in waiting to get consistency?

Re: GPT Unicorn has drawn a unicorn

#82

Post by flanked-evergl »

cvzakharchenko wrote: post #80

I gave the SVG unicorn back to GPT-4 and asked it, what is it. It recognized a head, eyes, body and legs. But it didn't recognize the unicorn. https://chat.openai.com/share/5409b417-b883-429f-893e-abe3d6...

You need to have quite a bit of imagination to say that is a unicorn, but less than you would need for other days.

Re: GPT Unicorn has drawn a unicorn

#83

Post by davrosthedalek »

luc4sdreyer wrote: post #44
pietroppeter wrote: post #31

Earlier quoted context omitted.

Is there some reference or explanation on why the model it is non deterministic at temperature 0?

I'm not aware of anything concrete by OpenAI, but others have offered possible explanations. One idea is that the cause is batched inference in sparse MoE (mixture of experts) models. https://152334h.github.io/blog/non-determinism-in-gpt-4/ HN discussion: https://news.ycombinator.com/item?id=37006224

So in some sense the spectre attack for AI?

Re: GPT Unicorn has drawn a unicorn

#84

Post by Klugistiono »

bhouston wrote: post #21
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.

The 'random noise' from a prompt "Draw a unicorn in svg" should still return Unicorns.

This is absolutly fine and it should start showing unicorn like drawings over a longer period and potentially finetuned ones over a longer period of time when the model changes.

Re: GPT Unicorn has drawn a unicorn

#86

Post by throwaway154 »

bhouston wrote: post #21
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.

Expecting a convergent series is bad logic.

Re: GPT Unicorn has drawn a unicorn

#87

Post by lewhoo »

We queried GPT-4 three times, at roughly equal time intervals over the span of a month while the system was being refined, with the prompt “Draw a unicorn in TikZ”. We can see a clear evolution in the sophistication of GPT-4’s drawings

Given how random GPT seems with what it's not designed to do the original research is really peculiar. Could it be that they queried GPT on three separate instances some n times and picked the best result ?

Re: GPT Unicorn has drawn a unicorn

#88

Post by kachnuv_ocasek »

hk__2 wrote: post #69
kachnuv_ocasek wrote: post #24

Earlier quoted context omitted.

It's using GPT-4, see the config file: https://github.com/adamkdean/gpt-unicorn/blob/357a466c366f67...

It’s using GPT-4 by default, but we can’t know what it uses for real since that’s in the environment config.

It says "gpt-4-0613" on the web page as well. Why would they pretend it's GPT-4 but use GPT-3.5 in the background?

Re: GPT Unicorn has drawn a unicorn

#89

Post by imtringued »

rav wrote: post #49
pietroppeter wrote: post #31

Earlier quoted context omitted.

Is there some reference or explanation on why the model it is non deterministic at temperature 0?

One important source of non-determinism is from using massive parallelism together with floating point arithmetic. In real math, a sum of numbers has an exact value that doesn't change if you change which order the numbers are added up in, but floating point arithmetic addition is not associative in the same way as real math, and parallelism can cause numbers to be added in a different order from execution to executi…

>and parallelism can cause numbers to be added in a different order from execution to execution

Parallelism doesn't magically add non-determinism of this kind unless you intentionally build it to be non deterministic. Nothing prevents you from processing an array in order in parallel.

Post reply on HN