Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT Unicorn has drawn a unicorn

gpt-unicorn.adamkdean.co.uk

Post reply on HN
Print view View original

71–80 of 207 posts

Re: GPT Unicorn has drawn a unicorn

#71

Post by brabel »

bhouston wrote: post #21
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.

I believe the logic is fine. You seem to think you need multiple data points from the same version of the model (i.e. multiple samples per day at least) would be necessary to judge the actual performance on each particular day.

That's worse logic. How would you visualize the very large sample you would get? Even with the current 118 samples (one per day) it's already difficult to find a pattern.

Would you "average" the samples?? That would not help IMO, you would need to average the score of each image, which requires either manually doing it or finding a reliable algorithm to do it automatically, but good luck with that.

So, a sample per day which allows clearly visualizing any change in the results over months and years is a valuable thing to do and I find it hard to improve on the methodology. You just need to keep in mind that one single picture from the sample is not enough, no one is going to disagree with that... but that doesn't make it "bad logic" and it's pretty thoughtless to say so.

Re: GPT Unicorn has drawn a unicorn

#72

Post by addandsubtract »

bhouston wrote: post #48
calderknight wrote: post #38

Earlier quoted context omitted.

I don't see the bad logic.

He said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for…

There's a linked blog post[0] that goes more into the methodology and reasoning.

"As mentioned in the hacker news discussion, the model doesn't change daily. [...] As OpenAI releases incremental updates, we'll see the model change automatically and be able to judge outputs. A single sample per day leads to quite different results, but that's fine I think. What I expect to see a year from now is an evolution of output. In variance: how varied are the outputs over a month?"

So yes, they could just produce 100 images with each new model release, but chose to spread those out over 1 per day instead. Is it the most scientific way to measure progress? No. Is it more fun and interesting to check back daily? Probably.

[0] https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploratio...

Re: GPT Unicorn has drawn a unicorn

#73

Post by throwuwu »

CrzyLngPwd wrote: post #43

There's no progress from something that is definitely not a unicorn towards a unicorn, the images seem randomly bad. I think it would be a lot of fun to give it the previous unicorn SVG attempt and ask it to make it more like a unicorn.

Read the paper https://arxiv.org/pdf/2303.12712.pdf

Page 7

Re: GPT Unicorn has drawn a unicorn

#74

Post by brabel »

amelius wrote: post #34
nullpilot wrote: post #2

For a second I thought this was about a GPT based product hitting a certain valuation, but I'm much more entertained by the actual content. This is great.

For a second I thought this was about a GPT based agent building a company that turned into a Unicorn.

I thought the same! That would be something.

Re: GPT Unicorn has drawn a unicorn

#77

Post by sigmoid10 »

rcme wrote: post #68
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

I'm not sure what you mean by random sampling. If I sample a random SVG, I wouldn't expect to look like anything, let alone roughly like a unicorn.

I mean random sampling in the sense how autoregressive language models like GPT generate sequences using token probabilities. It's not a random svg, but the text sequence that is used to draw it suffers from inherent non-determinism in the underlying model.

Re: GPT Unicorn has drawn a unicorn

#78

Post by itissid »

One thing people do/did with search engines was to observe their ranking stability over time on common queries like "Cats" or "Dogs". If there was any change one could meaningfully investigate what has changed.

Though this seems a bit more like neuropsychological eval by asking a blackbox(AI) questions.

Re: GPT Unicorn has drawn a unicorn

#79

Post by sigmoid10 »

MauranKilom wrote: post #52
sigmoid10 wrote: post #15

It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…

Yes, nothing about GPT4 changed today . But that's not the goal of the project (although I can't speak for the intentions of the submitter here). Currently there are two different GPT4 models represented in the samples, with quite significant quality difference between them. The quality (and variance in quality within a single model!) is interesting to see in such a comparison.

This variance also exists among outputs from the same model. Just scroll down a bit and you'll see drastic quality differences with the exact same model.
Post reply on HN