121–130 of 207 posts
Earlier quoted context omitted.
> This comment section is a super fascinating case study on the inherent flaws in human cognition. Especially when it comes to seeing patterns in random noise. The fact that some people believe that the model really has to have changed in the past few days is amazing You need only to look at the discourse around the Tesla FSD superusers to see this: they report a glitch at an intersection one day, then believe the ne…
Thats even worse for autonomous cars, there is so such data and noise there is no way to reproduce the issue, it's complete chaos. Whereas with a LLM if we control the seed we can 100% reproduce the same result
No, you can't. For the latest GPT models and the way they are run, this doesn't work anymore, making the experiment completely illogical. Some of the reasons are explained here pretty well: https://152334h.github.io/blog/non-determinism-in-gpt-4/
This is why, as a product manager, you should always test 20 hypotheses per month. At p-value of 0.05 this basically guarantees a successful product feature test every month!
For a second I thought this was about a GPT based product hitting a certain valuation, but I'm much more entertained by the actual content. This is great.
I can't find the Twitter account that was running the "Follow everything GPT says to make $1m" but for a second I thought it was going to be that.
This is why, as a product manager, you should always test 20 hypotheses per month. At p-value of 0.05 this basically guarantees a successful product feature test every month!
Earlier quoted context omitted.
We’re pretty sure the nondeterminism is batching + mixture of experts + contention for specific experts
If by batching you mean bad code that fails to sort or relies on hardware to best-guess how things sort, then sure, that's called a bug. Also, "we're pretty sure" is rather self-important while also admitting total, abject failure to produce a deterministic result. You shouldn't blame yourself. A lot of people had the same feeling after staking their life on the revolutionary properties of NFTs.
Even if it's not sparse MoE, chances are high that the non-determinism is introduced somewhere purely as a performance optimization. The article speculates that OpenAI knows this well and hides it to protect the model internals.
[1] https://152334h.github.io/blog/non-determinism-in-gpt-4/
I would argue that `image-2023-04-25` looks comparable close to a unicorn. The art style does vary a lot though.
Earlier quoted context omitted.
Thats even worse for autonomous cars, there is so such data and noise there is no way to reproduce the issue, it's complete chaos. Whereas with a LLM if we control the seed we can 100% reproduce the same result
>> if we control the seed we can 100% reproduce the same result No, that's the problem. You can't. You should be able to, but you can't. If you could, they wouldn't be scary. But we have Temperature Zero, different results. Because no one gave enough of a shit when coding them, and no one gives enough of a shit to try to fix the issue . This is what in any other industry would be called gross negligence.
For a second I thought this was about a GPT based product hitting a certain valuation, but I'm much more entertained by the actual content. This is great.
For a second I thought this was about a GPT based agent building a company that turned into a Unicorn.