Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-determinism in GPT-4 is caused by Sparse MoE

152334h.github.io

Post reply on HN
Print view View original

91–100 of 186 posts

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#91

Post by dwpdwpdwpdwpdwp »

jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

Mathematically, computation is deterministic. The author dismisses or ignores the many ways that the physical apparatus driving the computation can force the result of a software application to be a function of time.

Calling GetTimeOfDay() could do it.

Clock frequency drift between multiple processors could it.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#92

Post by xyzzy_plugh »

n2d4 wrote: post #82
jiggawatts wrote: post #77

Earlier quoted context omitted.

Read the article you linked. It literally says that the GPU is deterministic, the NVIDIA libraries on top are deterministic, but it is Tensorflow that introduces variability (errors!) for “performance”. My argument is that it is the AI/ML code that is introducing non-determinism, usually by sacrificing repeatability to gain performance. That's precisely what's happening here. Tensorflow introduced a "harmless"[1] dat…

As the article says, cuBLAS is deterministic, but other CUDA primitives (eg. some of those in cudNN) are not. Yes, the non-determinism is being introduced somewhere, but that is splitting hairs. The point is that the primitives that you work with on GPUs are non-deterministic by design. I mostly take issue with you phrasing it as a bug and using it to insult the authors.

How is that splitting hairs?

> The point is that the primitives that you work with on GPUs are non-deterministic by design.

This is just blatantly wrong. There are _some_ operations that can be non-deterministic in some scenarios but they are not necessary.

GPUs are deterministic. If you ask them to add a million floats in order, you get the same result every time. If you ask them to add a million floats in some arbitrary order, then you may get different results every time. The distinction is that someone had to ask the GPU to do that. It's a choice.

> I mostly take issue with you phrasing it as a bug and using it to insult the authors.

It's a bug, whether it insults the authors or not is irrelevant. It's most definitely a bug.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#93

Post by neatze »

jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

hmm, how, I wonder if Alhazen’ s Circular Billiard Problem[1] results for n steps in simulation will be same for multiple runs.

[1] https://forumgeom.fau.edu/FG2012volume12/FG201216.pdf

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#94

Post by water9 »

dwpdwpdwpdwpdwp wrote: post #91
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

Mathematically, computation is deterministic. The author dismisses or ignores the many ways that the physical apparatus driving the computation can force the result of a software application to be a function of time. Calling GetTimeOfDay() could do it. Clock frequency drift between multiple processors could it.

When theory fails to consult reality.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#96

Post by armchairhacker »

dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

I read the abstract and look at the pretty figures :)

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#97

Post by refulgentis »

goodside wrote: post #39
refulgentis wrote: post #26

Earlier quoted context omitted.

Topic laundering, the probabilities are the probabilities, you don't suddenly get wrong probabilities with more training on more data

You do, because it’s not just more training it’s PPO updates instead of MLE. It’s no longer trying to estimate the token distribution of the training corpus, it’s trying to shift logprobs into tokens that maximize expected reward from the RM. The GPT-4 technical report has a figure showing that logprobs become less well calibrated as confidence scores in the RLHF vs pre-train model.

Fascinating, ty

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#98

Post by refulgentis »

phillipcarter wrote: post #45
derwiki wrote: post #7

Earlier quoted context omitted.

GPT4 web chat for two hours a day? I buy that. Using the API repeatedly for the same inputs, eg developing a program, and the non-determinism is hard to miss.

Yeah, it's one of the first things you notice when trying to do some kind of "feed GPT some data and get it to produce a novel answer to a question" task with the API.

No, because if you wanted a novel answer, why would you set 0 temperature? ;)

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#99

Post by refulgentis »

dekhn wrote: post #11
sebzim4500 wrote: post #9

Earlier quoted context omitted.

I would imagine that most people use nonzero temperature, so they won't need to look for any explanation for non-determinism.

Literally the first thing I did when I had llama.cpp working was set the temperature to 0 and repeat queries. (but that's mainly because I'm a weird old scientist with lots of experience with nondeterminism in software).

I did too, Kmeans broke me a couple years ago: but, never temperature at 0 with long length, and trusted my instinct instead of actual diffs. This is was the first time I actually diffed

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#100

Post by n2d4 »

xyzzy_plugh wrote: post #90
n2d4 wrote: post #78

Earlier quoted context omitted.

I just edited my comment while you were writing your comment to add an explanation. The point here is that some primitives in eg. cudNN are non-deterministic. Whether you classify that as a race condition or not is a different question; but it's intended behaviour.

Right but that's not an inherent GPU determinism issue. It's a software issue. https://github.com/tensorflow/tensorflow/issues/3103#issueco... is correct that it's not necessary, it's a choice. Your line of reasoning appears to be "GPUs are inherently non-deterministic don't be quick to judge someone's code" which as far as I can tell is dead wrong. Admittedly there are some cases and instructions that may result in…

What I'm saying is "there are non-deterministic primitives", not "there are no deterministic primitives".
Post reply on HN