Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-determinism in GPT-4 is caused by Sparse MoE

152334h.github.io

Post reply on HN
Print view View original

101–110 of 186 posts

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#101

Post by cypress66 »

brmgb wrote: post #44
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

> I noticed people from hacker news routinely read scientific papers. Highly doubt that. It’s very hard to actually read scientific papers when you are not actively doing research. You can’t just read a research paper in isolation. It’s next to useless. You need to understand its context, where it stands with regard to its sources and what it brings which is actually new and valuable. It’s nearly impossible to do pro…

You don't need to be doing research to read an ML paper. With some general knowledge in AI you should be able to understand most papers.

And even then, sometimes you don't understand or care about their procedures, and you just want to look at the pretty results (check out this song they generated using AI!). There's even a very popular YouTube channel that focuses on this (two minute papers).

Finally, you usually hear about these cool papers via Twitter / X

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#102

Post by xyzzy_plugh »

n2d4 wrote: post #70
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

You've moved the goal posts. You're conflating CUDA with GPUs. From Wikipedia:

> CUDA (or Compute Unified Device Architecture) is a proprietary and closed source parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing units (GPUs) for general purpose processing, an approach called general-purpose computing on GPUs (GPGPU). CUDA is a software layer that gives direct access to the GPU's virtual instruction set and parallel computational elements, for the execution of compute kernels.

Is the issue we're discussing because of the GPU or is it because of choices made in software libraries?

The parent is right, there is a deterministic, reproducible way to solve these problems, so if determinism is a desired or expected property, then this is a bug. It's not an inherent problem like you make it out to be. The fact that "workarounds" are given in what you link prove this.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#103

Post by cainxinth »

I asked GPT to explain this:

>In the MoE approach, different "experts" or portions of the model are selected for different parts of the input data. The selection of which experts to use can be influenced by several factors, including the specific content of the input data, the order in which data is processed in a batch, and possibly even minor variations in the internal state of the model.

>This "expert selection" process introduces a level of stochasticity, or randomness, into the model's operation. For example, if you process the same input data twice in slightly different contexts (e.g., as part of different batches), you might end up consulting slightly different sets of experts, leading to slightly different outputs.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#104

Post by xyzzy_plugh »

n2d4 wrote: post #100
xyzzy_plugh wrote: post #90

Earlier quoted context omitted.

Right but that's not an inherent GPU determinism issue. It's a software issue. https://github.com/tensorflow/tensorflow/issues/3103#issueco... is correct that it's not necessary, it's a choice. Your line of reasoning appears to be "GPUs are inherently non-deterministic don't be quick to judge someone's code" which as far as I can tell is dead wrong. Admittedly there are some cases and instructions that may result in…

What I'm saying is "there are non-deterministic primitives", not "there are no deterministic primitives".

Yes, and `gettimeofday` is a non-deterministic primitive. There is nothing special about GPUs here. If you write tests that fail sometimes because you used non-deterministic primitives like gettimeofday and someone files a bug we don't throw up our hands and say "this is not a bug but due to how CPUs work." We remove the non-deterministic bit.

There's no difference here. This isn't a GPU problem.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#105

Post by CSMastermind »

nerdponx wrote: post #18
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

> I noticed people from hacker news routinely read scientific papers. Do they? I suspect that most don't, and those that do are either in specialized careers or are engaged in some kind of scientific research. Some interesting research gets disseminated via Twitter and chatrooms. Or maybe you follow a podcast that mentions new research. But you might also be following new publications from a handful of reputable jour…

I typically look up and read a paper when it's referenced in discussion or cited in something else, I'm reading/watching, and the purported contents seem surprising to me. This normally happens 3 or 4 times a week.

Honestly many papers are written in a way that's hard to approach and difficult to understand unless you're prepared to reread them a few times.

You're better off just getting your science news from actual science communicators and not the raw source.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#106

Post by famouswaffles »

albystein wrote: post #46
gojomo wrote: post #34

Not sure I understand the excerpt from the referenced paper. Is it saying that part of its more-efficient inferencing relies on mixing tokens from completely-separate inputs – eg, from other users? And then, depending on what other inputs chance into the same grouping, the relative assignment-to-'experts' varies, and thus the eventual completions? If so, I'd see that as not just introducing non-determinism, but also…

this seems like a plausible outcome, and if true could spell disaster for OpenAI models relative to the competition and open source models. Currently, reliability is one of the core obstacles preventing widespread adoption of LLMs in many business critical workflows. And if these rumors, that GPT-4 is inherently un-deterministic and unreliable, are true then most enterprises are better off finetuning open source LLMs…

>as domain-specific models will always outperform generalist ones

This is not even close to true for Language models.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#107

Post by MacsHeadroom »

nerdponx wrote: post #18
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

> I noticed people from hacker news routinely read scientific papers. Do they? I suspect that most don't, and those that do are either in specialized careers or are engaged in some kind of scientific research. Some interesting research gets disseminated via Twitter and chatrooms. Or maybe you follow a podcast that mentions new research. But you might also be following new publications from a handful of reputable jour…

Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week.

I probably averaged 20 a week back in March when open source AI was booming in the wake of Llama and on the heels of GPT-4.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#108

Post by jes5199 »

n2d4 wrote: post #70
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

it’s basically always reasonable to insult someone’s code because we are computer programmers and we know what we have done

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#109

Post by cpgxiii »

xyzzy_plugh wrote: post #104
n2d4 wrote: post #100

Earlier quoted context omitted.

What I'm saying is "there are non-deterministic primitives", not "there are no deterministic primitives".

Yes, and `gettimeofday` is a non-deterministic primitive. There is nothing special about GPUs here. If you write tests that fail sometimes because you used non-deterministic primitives like gettimeofday and someone files a bug we don't throw up our hands and say "this is not a bug but due to how CPUs work." We remove the non-deterministic bit. There's no difference here. This isn't a GPU problem.

Except the issue is inextricably linked to GPUs. All of the work in practical DNNs exists because of the extreme parallel performance available from GPUs, and that performance is only possible with non-deterministic threading. You can't get reasonable training and inference time on existing hardware without it.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#110

Post by benreesman »

n2d4 wrote: post #70
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

I don’t know about how insulting it is, I don’t like rushing things out but we’ve all had to.

People are rushing like crazy to get there first with X for AI all over the place, it would be pretty shocking if there weren’t wires sticking out everywhere.

I don’t think that says anything positive or negative about the hackers involved.

Post reply on HN