Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-determinism in GPT-4 is caused by Sparse MoE

152334h.github.io

Post reply on HN
Print view View original

141–150 of 186 posts

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#141

Post by xiphias2 »

cpgxiii wrote: post #123
d0mine wrote: post #115

Earlier quoted context omitted.

1000 threads can run in parallel. It doesn't prevent us to sum their results deterministically: results = ThreadPool(workers=1000).imap_unordered(calc, inputs) print(math.fsum(results)) Due to the magic of the fsum alg, the result is deterministic whatever order we get results in. https://docs.python.org/3/library/math.html#math.fsum

That's not the operation being performed on GPUs that is the problem. The issue is that fundamentally GPUs allow for high performance operations using atomics, but this comes at the cost of nondeterministic results. You can get deterministic results but doing so comes with a significant performance costs.

Using atomics is easier than warp operations (using warp shuffle for example), but warp shuffle is quite fast.

I guess if determinism is so important implementations can be changed, it is just maybe not that high priority.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#142

Post by hyperthesis »

ShamelessC wrote: post #120
hyperthesis wrote: post #81

MoE: Mixture of Experts

There’s a comment that’s 3 hours older than yours that clarifies this.

I searched for MoE in the comments and didn't see it. ah, you must mean this one https://news.ycombinator.com/item?id=37006549, which doesn't include "MoE", so that's why I didn't find it. Still, my comment's upvotes show it was helpful to some - maybe they searched for "MoE" too, instead of "mixture of experts".

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#143

Post by brmgb »

cypress66 wrote: post #101
brmgb wrote: post #44

Earlier quoted context omitted.

> I noticed people from hacker news routinely read scientific papers. Highly doubt that. It’s very hard to actually read scientific papers when you are not actively doing research. You can’t just read a research paper in isolation. It’s next to useless. You need to understand its context, where it stands with regard to its sources and what it brings which is actually new and valuable. It’s nearly impossible to do pro…

You don't need to be doing research to read an ML paper. With some general knowledge in AI you should be able to understand most papers. And even then, sometimes you don't understand or care about their procedures, and you just want to look at the pretty results (check out this song they generated using AI!). There's even a very popular YouTube channel that focuses on this (two minute papers). Finally, you usually he…

> You don't need to be doing research to read an ML paper. With some general knowledge in AI you should be able to understand most papers.

I have a degree which involved reading some ML papers and I seriously doubt that. The field is flooded with papers which looks good when you quickly read them but are actually worthless because they misrepresent the state of the art or intentionally don’t compare their methods with other papers they should know.

> And even then, sometimes you don't understand or care about their procedures, and you just want to look at the pretty results

That’s fair but I wouldn’t call that reading a scientific paper.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#144

Post by PeterisP »

zx14 wrote: post #139
n2d4 wrote: post #70

Earlier quoted context omitted.

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

There isn't much of a culture around code quality in ML / AI / DS.

It's not a code quality issue, there are ways to ensure determinism (sometimes you just need to set a flag), however, they are intentionally explicitly not used in order to gain performance.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#145

Post by PeterisP »

f1shy wrote: post #127

I see in the comments it seems to be a huge miss understanding between 2 uses of “non-deterministic”: 1) from normal English: cannot be determined beforehand (results may vary) 2) from theory of computation: loosely “parallel computation” (unknown path to the solution)

For floating point math, there's no distinction, as "parralel computation with unknown path to the solution" inherently implies "results will vary", as (a+b)+c != a+(b+c).

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#146

Post by namibj »

DeathArrow wrote: post #132
KolenCh wrote: post #113

Earlier quoted context omitted.

What you said can be violated when parallelism is involved. One such example is that we know some floating point operations such as addition and multiplication are non-commutative, hence it depends on order of execution to complete reduction for example. And then in parallel situation, some implementation will make the order or reduction non-deterministic (for performance reason) and hence the final result also non-d…

It's still deterministic even if the results appear not to be. If you have memory, CPU cache, CPU registers in the same state, you will get the very same results. You need a source of entropy for the results to be non deterministic.

Actually, clock domain crossing for asynchronous clocks (as is AFAIK typical for granular dynamic frequency scaling, like running CPU cores at individual frequencies instead of all at the same, because it quite softly smoothes over to any new target frequency to prevent glitches) implicitly includes thermal noise in the raw transistors that determine which of the two involved clock edges happened earlier (a decision that eventually ends up truly random when they are at (almost) exactly the same time). And this is involved in even L3 hit latency.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#147

Post by moonchrome »

cratermoon wrote: post #125

> It’s well-known at this point that GPT-4/GPT-3.5-turbo is non-deterministic, even at temperature=0.0 Interestingly, on another discussion there was a claim that setting the temperature to 0.0 made gpt-4 deterministic: https://news.ycombinator.com/item?id=36503146

This guy probably never did anything nontrivial with the API - you notice almost instantly that the chat models (both 3.5 and 4) are nondeterministic at 0 temperature. Source - built a documentation search bot and had it crap out on me on copy pasted prompts when I was demoing it.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#148

Post by [deleted] »

jiggawatts wrote: post #77
n2d4 wrote: post #70

Earlier quoted context omitted.

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

Read the article you linked. It literally says that the GPU is deterministic, the NVIDIA libraries on top are deterministic, but it is Tensorflow that introduces variability (errors!) for “performance”. My argument is that it is the AI/ML code that is introducing non-determinism, usually by sacrificing repeatability to gain performance. That's precisely what's happening here. Tensorflow introduced a "harmless"[1] dat…

[deleted]

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#149

Post by dlkf »

spott wrote: post #119
xyzzy_plugh wrote: post #92

Earlier quoted context omitted.

How is that splitting hairs? > The point is that the primitives that you work with on GPUs are non-deterministic by design. This is just blatantly wrong. There are _some_ operations that can be non-deterministic in some scenarios but they are not necessary. GPUs are deterministic. If you ask them to add a million floats in order, you get the same result every time. If you ask them to add a million floats in some arbi…

Basically any parallel map-reduce operation using non-commutative reduce operators[0] is non-deterministic unless you specifically sort after/during the gather, or block on the gather (and gather to a thread-determined memory location). Sorting and blocking takes time. If you remove the sort/block, you will get a non-deterministic answer when operating on floats for a wide variety of reduce operations, but it will be…

Your comment, along with cpgxiii and n2d4’s are all really good. I have a question: suppose training and inference of an LLM were made to be deterministic at the cost of performance.

Would the cost be “everything will take twice as long” or would it be more like “inference will take a week and training will take a couple lifetimes”?

If it’s the latter, then it seems disingenuous to call this a “bug.” It’s like saying F1 cars could be horse drawn, and they only use internal combustion for “performance reasons.” If its the former, then maybe there is a more interesting discussion to be had about the potential benefits of determinism? (That said, I agree with n2d4 that it’s stupid to insult the authors. Talk is cheap and building is hard.)

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#150

Post by mst »

TechBro8615 wrote: post #63
pazimzadeh wrote: post #6

Mixture of Experts

Thanks. I assumed it was Margin of Error. The article doesn't expand the acronym until midway through the post, where it appears almost accidentally. Perhaps the intended audience is a mixture of experts, of which I'm not a part.

I suspect the article is written primarily to be clear to people sufficiently immersed in the relevant areas to be able to have a concrete opinion on the theory.

Also I strongly suspect that at least in the case of -me-, an article that was easier for me to understand wouldn't make the underlying theory any easier for me to judge.

(on the upside, at least I -did- understand and appreciate your self deprecating pun :)

Post reply on HN