Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT Unicorn has drawn a unicorn

gpt-unicorn.adamkdean.co.uk

Post reply on HN
Print view View original

201–207 of 207 posts

Re: GPT Unicorn has drawn a unicorn

#201

Post by joshspankit »

kozikow wrote: post #176
Karawebnetwork wrote: post #159

Earlier quoted context omitted.

I noticed similar a behavior in Stable Diffusion forums where people believe that the model they downloaded and are running offline is getting better at understanding their prompts.

Stable diffusion most likely don't do it, but even static model that as an input takes embedding of all your historic prompts + current prompt would get progressively give you better inputs as you use it.

Yes, but currently that would be a conscious choice (and extra intentional effort)

Re: GPT Unicorn has drawn a unicorn

#202

Post by fsloth »

jquery wrote: post #132
fsloth wrote: post #62

Earlier quoted context omitted.

I don't think anyone claims this is an iterative linear measure, rather than a step function. SVG can present arbitrarily complex graphics. The underlying display tech supports what ever fidelity GPT will eventually mature into. Has GPT plateaud? Will it be stuck forever at this hilariously naive level of competence at SVG art? Will it mature into Midjourney level competence? I have no frigging clue. Since the token…

I mean, how many humans can draw art by writing out svg? If that's not in the training set, I don't even see how GPT-4 gets much better at this over time.

GPT-4 is really great at transferring concepts between domains.

That's one the reasons why GPT when it works, feels magical. SVG art does not need to be in it's training set, as long as it knows how to present geometric concepts in SVG.

A good unicorn would require capabilities something like "the outline of unicorn is composed of lines {...}." -> "export lines as svg".

Re: GPT Unicorn has drawn a unicorn

#203

Post by jquery »

fsloth wrote: post #202
jquery wrote: post #132

Earlier quoted context omitted.

I mean, how many humans can draw art by writing out svg? If that's not in the training set, I don't even see how GPT-4 gets much better at this over time.

GPT-4 is really great at transferring concepts between domains. That's one the reasons why GPT when it works, feels magical. SVG art does not need to be in it's training set, as long as it knows how to present geometric concepts in SVG. A good unicorn would require capabilities something like "the outline of unicorn is composed of lines {...}." -> "export lines as svg".

Ok that makes perfect sense, thanks.

Re: GPT Unicorn has drawn a unicorn

#204

Post by olabyne »

noduerme wrote: post #118
olabyne wrote: post #113

Earlier quoted context omitted.

Thats even worse for autonomous cars, there is so such data and noise there is no way to reproduce the issue, it's complete chaos. Whereas with a LLM if we control the seed we can 100% reproduce the same result

>> if we control the seed we can 100% reproduce the same result No, that's the problem. You can't. You should be able to, but you can't. If you could, they wouldn't be scary. But we have Temperature Zero, different results. Because no one gave enough of a shit when coding them, and no one gives enough of a shit to try to fix the issue . This is what in any other industry would be called gross negligence.

That's a OpenAi problem, not a LLM problem

Re: GPT Unicorn has drawn a unicorn

#205

Post by olabyne »

sigmoid10 wrote: post #122
olabyne wrote: post #113

Earlier quoted context omitted.

Thats even worse for autonomous cars, there is so such data and noise there is no way to reproduce the issue, it's complete chaos. Whereas with a LLM if we control the seed we can 100% reproduce the same result

>Whereas with a LLM if we control the seed we can 100% reproduce the same result No, you can't. For the latest GPT models and the way they are run, this doesn't work anymore, making the experiment completely illogical. Some of the reasons are explained here pretty well: https://152334h.github.io/blog/non-determinism-in-gpt-4/

This sounds more like a bug than a feature.

Re: GPT Unicorn has drawn a unicorn

#206

Post by noduerme »

jquery wrote: post #129
noduerme wrote: post #118

Earlier quoted context omitted.

>> if we control the seed we can 100% reproduce the same result No, that's the problem. You can't. You should be able to, but you can't. If you could, they wouldn't be scary. But we have Temperature Zero, different results. Because no one gave enough of a shit when coding them, and no one gives enough of a shit to try to fix the issue . This is what in any other industry would be called gross negligence.

A lot of stuff behind the scenes is going on to batch and route queries to GPT-4 models that are in perturbed states already[1]. This isn't gross negligence, this is basic capitalism. If you want sole access to a GPT-4 MoE cluster starting fresh, it's gonna cost you. 1. https://152334h.github.io/blog/non-determinism-in-gpt-4/

Interesting article. I can see how it makes sense for OpenAI or someone with a LLM to take advantage of any entropy that presented itself, as a shortcut to non-repetitive answers. I'm not sure if you're saying that these LLMs take on new characteristics as they get more randomized? Or just that it would be hard to get your hands on a fresh one to test the determinism of?

Re: GPT Unicorn has drawn a unicorn

#207

Post by sigmoid10 »

rcme wrote: post #172
sigmoid10 wrote: post #77

Earlier quoted context omitted.

I mean random sampling in the sense how autoregressive language models like GPT generate sequences using token probabilities. It's not a random svg, but the text sequence that is used to draw it suffers from inherent non-determinism in the underlying model.

Relying on token probabilities seems like the exact opposite of random

The neural network just generates a set of probabilities for all tokens. The actual next token is then sampled from this set, which is always random for T>0 (and in the case of GPT4 even for T=0, because of the way the model itself works).
Post reply on HN