Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-determinism in GPT-4 is caused by Sparse MoE

152334h.github.io

Post reply on HN
Print view View original

131–140 of 186 posts

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#131

Post by DeathArrow »

n2d4 wrote: post #78
Tunabrain wrote: post #76

Earlier quoted context omitted.

GPUs are deterministic machines, even for floating point. The behavior in the linked article has to do with the use of atomic adds to reduce sums in parallel. Floating point addition is not associative, so the order in which addition occurs matters. When using atomic adds this way, you get slightly different results depending on the order in which threads arrive at the atomic add call. It's a simple race condition, a…

I just edited my comment while you were writing your comment to add an explanation. The point here is that some primitives in eg. cudNN are non-deterministic. Whether you classify that as a race condition or not is a different question; but it's intended behaviour.

If the hardware is deterministic, so are the results. You can't generate random numbers purely in software with deterministic hardware.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#132

Post by DeathArrow »

KolenCh wrote: post #113
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

What you said can be violated when parallelism is involved. One such example is that we know some floating point operations such as addition and multiplication are non-commutative, hence it depends on order of execution to complete reduction for example. And then in parallel situation, some implementation will make the order or reduction non-deterministic (for performance reason) and hence the final result also non-d…

It's still deterministic even if the results appear not to be. If you have memory, CPU cache, CPU registers in the same state, you will get the very same results. You need a source of entropy for the results to be non deterministic.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#134

Post by toxik »

KolenCh wrote: post #113
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

What you said can be violated when parallelism is involved. One such example is that we know some floating point operations such as addition and multiplication are non-commutative, hence it depends on order of execution to complete reduction for example. And then in parallel situation, some implementation will make the order or reduction non-deterministic (for performance reason) and hence the final result also non-d…

Minor nit but commutative is the wrong term. Floats always obey a+b == b+a, but not associativity: (a+b)+c != a+(b+c).

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#135

Post by WithinReason »

DeathArrow wrote: post #131
n2d4 wrote: post #78

Earlier quoted context omitted.

I just edited my comment while you were writing your comment to add an explanation. The point here is that some primitives in eg. cudNN are non-deterministic. Whether you classify that as a race condition or not is a different question; but it's intended behaviour.

If the hardware is deterministic, so are the results. You can't generate random numbers purely in software with deterministic hardware.

The behaviour of atomic operations is definitely not deterministic. E.g. if you have a lot of atomic adds, every time you run the code you'll get a different result without a random number generator.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#136

Post by WithinReason »

d0mine wrote: post #115
cpgxiii wrote: post #109

Earlier quoted context omitted.

Except the issue is inextricably linked to GPUs. All of the work in practical DNNs exists because of the extreme parallel performance available from GPUs, and that performance is only possible with non-deterministic threading. You can't get reasonable training and inference time on existing hardware without it.

1000 threads can run in parallel. It doesn't prevent us to sum their results deterministically: results = ThreadPool(workers=1000).imap_unordered(calc, inputs) print(math.fsum(results)) Due to the magic of the fsum alg, the result is deterministic whatever order we get results in. https://docs.python.org/3/library/math.html#math.fsum

That summation is slow and would not be used in practice.

You could use just one thread on your 10000 thread GPU too and it would be deterministic, sure. Completely beside the point.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#137

Post by LudwigNagasena »

allisdust wrote: post #121
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

Don't read them for the sake of reading them. Read them to solve your current problem or trying to keep up with advancements in a narrow field you love. Most papers (especially the ones in deep learning) seem to also have a mathematical fetish (to put it mildly) where needless representations are used where none are required and are self evident (for example inputs belong to Real number set). It ends up making the pa…

> Most papers (especially the ones in deep learning) seem to also have a mathematical fetish (to put it mildly) where needless representations are used where none are required and are self evident (for example inputs belong to Real number set). It ends up making the paper pseudo complex and unapproachable.

I completely disagree with that. Spelling out math is literally something out of 12th century. It just hinders understanding, if you have basic STEM-level math literacy, which anyone who reads an ML paper is implied to have (how could you seriously study linear algebra and calculus without it?).

Math may actually be the first thing you recognise in a paper, which can help you cross-reference the text to understand it.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#138

Post by WithinReason »

DeathArrow wrote: post #130
n2d4 wrote: post #70

Earlier quoted context omitted.

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

So you can generate true random numbers using just the GPU parallelism? Consider me impressed!

Yes you can, and it's been done:

https://link.springer.com/article/10.1007/s11071-015-2287-7

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#139

Post by zx14 »

n2d4 wrote: post #70
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

There isn't much of a culture around code quality in ML / AI / DS.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#140

Post by WithinReason »

DeathArrow wrote: post #132
KolenCh wrote: post #113

Earlier quoted context omitted.

What you said can be violated when parallelism is involved. One such example is that we know some floating point operations such as addition and multiplication are non-commutative, hence it depends on order of execution to complete reduction for example. And then in parallel situation, some implementation will make the order or reduction non-deterministic (for performance reason) and hence the final result also non-d…

It's still deterministic even if the results appear not to be. If you have memory, CPU cache, CPU registers in the same state, you will get the very same results. You need a source of entropy for the results to be non deterministic.

Sure, but they will never be in the same state, which can even be used as a source of entropy: https://link.springer.com/article/10.1007/s11071-015-2287-7
Post reply on HN