Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-determinism in GPT-4 is caused by Sparse MoE

152334h.github.io

Post reply on HN
Print view View original

151–160 of 186 posts

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#151

Post by mst »

dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

For computer science, https://blog.acolyer.org/ is called The Morning Paper and talks about one interesting paper per post.

Edit: It seems to've gone on indefinite hiatus but there's a lot of backlog already there and some of it's really quite fascinating.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#152

Post by mst »

jldugger wrote: post #112
puzzledobserver wrote: post #51

Earlier quoted context omitted.

I'd like to disagree with this. In particular, about [1]: It is a collection of papers in many different topics. There is little technical overlap between Alan Turing's Entscheidungsproblem paper, for instance, and Hoare's paper on axiomatic semantics. Also, the papers are all from the 70s. They're uniformly influential papers, and have shaped the field, but the fields and the vernacular used by working researchers i…

I feel like you're giving advice on how to become a PhD student, and frankly, that's not the point of the question, and if it is: any grad student who can't read papers should ask their advisor for advice. So I take OP's perspective to be from a practitioner (such as myself). Apart from my colleagues in R&D, we aren't called upon to write new papers that demands expertise in ever increasing narrowness. Instead we are…

Old ideas that were good but didn't become common/standard are something I run across a fair bit in papers and yeah, they're often way behind the state of the art but also a lot easier for me to understand/implement and far better than the relatively naive approach I'd've taken otherwise.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#153

Post by mst »

segfaultbuserr wrote: post #48
TechBro8615 wrote: post #35

Earlier quoted context omitted.

I don't mind reading research papers, but they're really annoying to read on a phone screen. I remember a few years ago, an HN comment shared a link to some tool that could convert a PDF to single column text and make it more readable on a phone screen, but I can't find it. Anyone remember this or have the link?

> but they're really annoying to read on a phone screen. +1. I've already read probably 100 research papers this year in search of solutions to some technical problems, mostly while lying on bed with a tablet. I won't read as much without it.

Once phones got relatively big (i.e. 'phablet' ceased to exist as a concept because that size was just 'phone' now) I switched to using a 7/8" tablet with my SIM in it as my primary portable device (Nexus 7 and now Galaxy Tab A6).

Means I have to carry it in my jacket pocket or a side pocket on my combats but the bigger phones weren't comfortable in my trousers' top pocket anyway so for me at least the trade-off is well worth it.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#154

Post by microtonal »

WanderPanda wrote: post #126
cpgxiii wrote: post #109

Earlier quoted context omitted.

Except the issue is inextricably linked to GPUs. All of the work in practical DNNs exists because of the extreme parallel performance available from GPUs, and that performance is only possible with non-deterministic threading. You can't get reasonable training and inference time on existing hardware without it.

In my experience cuBLAS is deterministic, since matmul is the most intensive part I don‘t see other reasons for non-determinism other than sloppyness (at least when just a single GPU is involved)

Yeah. In curated transformers [1] we are seeing completely deterministic output across multiple popular transformer architectures on a single GPU (there can be variance between GPUs due to different kernels). Of course, it completely depends on what ops and implementations you are using. But most transformers do not use ops that are typically non-deterministic to be fast (like scatter-add).

One non-determinism we see with a temperature of 0 is that once you have quantized weights, many predicted pieces will have the same probability, including multiple pieces with the highest probability. And then the sampler (if you are not using a greedy decoder) will sample from those pieces. So, generation is non-deterministic with a temperature of 0.

In other words, a temperature of 0 is a poor man’s greedy decoding. (It is totally possible that OpenAI’s implementation switches to a greedy decoder with a temperature of 0).

[1] https://github.com/explosion/curated-transformers

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#156

Post by xyzzy_plugh »

spott wrote: post #119
xyzzy_plugh wrote: post #92

Earlier quoted context omitted.

How is that splitting hairs? > The point is that the primitives that you work with on GPUs are non-deterministic by design. This is just blatantly wrong. There are _some_ operations that can be non-deterministic in some scenarios but they are not necessary. GPUs are deterministic. If you ask them to add a million floats in order, you get the same result every time. If you ask them to add a million floats in some arbi…

Basically any parallel map-reduce operation using non-commutative reduce operators[0] is non-deterministic unless you specifically sort after/during the gather, or block on the gather (and gather to a thread-determined memory location). Sorting and blocking takes time. If you remove the sort/block, you will get a non-deterministic answer when operating on floats for a wide variety of reduce operations, but it will be…

Nothing I said conflicts with this, though?

Yes, if you eschew determinism for the sake of raw performance then the result will be non-deterministic. But you don't have to do this, nor is it inherently untenable to solve these problems in a deterministic way.

Sure it may require some performance overhead, and increase development time, but it's no different than writing deterministic code elsewhere. It's disingenuous to hand-wave away the solution because of some opaque cost or overhead we're unwilling to entertain. None of the parent posts ever mention performance tradeoffs.

In particular there is no indication that the problem being discussed couldn't be solved with determinism in an equivalent amount of time. You're making my point: GPUs are deterministic, software may decide not to be.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#157

Post by xyzzy_plugh »

dlkf wrote: post #149
spott wrote: post #119

Earlier quoted context omitted.

Basically any parallel map-reduce operation using non-commutative reduce operators[0] is non-deterministic unless you specifically sort after/during the gather, or block on the gather (and gather to a thread-determined memory location). Sorting and blocking takes time. If you remove the sort/block, you will get a non-deterministic answer when operating on floats for a wide variety of reduce operations, but it will be…

Your comment, along with cpgxiii and n2d4’s are all really good. I have a question: suppose training and inference of an LLM were made to be deterministic at the cost of performance. Would the cost be “everything will take twice as long” or would it be more like “inference will take a week and training will take a couple lifetimes”? If it’s the latter, then it seems disingenuous to call this a “bug.” It’s like saying…

> That said, I agree with n2d4 that it’s stupid to insult the authors. Talk is cheap and building is hard.

If your code offers an expectation of determinism then it's sloppy to not distinguish where there isn't determinism. There's nothing difficult about writing a comment to the effect of "this function is non-deterministic. For deterministic results, use X".

The code is sloppy if the developers didn't consider determinism and offer nothing to consumers, or if the consumers writing software cannot know where non-determinism is introduced.

If that's somehow insulting then I'd say someone has very thin skin.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#158

Post by mschuster91 »

n2d4 wrote: post #70
jiggawatts wrote: post #64

Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…

No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…

> I would generally refrain from insulting other people's code if you don't know much about the system it's written on.

Well, the general state of how utterly shoddy most of the code in the AI/ML ecosystem is is observable to anyone trying to follow a guide on how to set up Stable Diffusion on AWS. It's a fucking mess of trying various combinations of driver versions, Ubuntu kernel versions, Python versions, and the fact that Python requirements.txt (similar to NodeJS) doesn't pin versions of transitive dependencies doesn't make it easier because it makes for very brittle and not reproducible builds/guides. Oh, and at least some of that stuff won't work without root.

Yeah I'll keep AI shit cordoned off in its own subnet.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#159

Post by famouswaffles »

visarga wrote: post #118
mrtranscendence wrote: post #114

Earlier quoted context omitted.

> And if these rumors, that GPT-4 is inherently un-deterministic and unreliable, are true then most enterprises are better off finetuning open source LLMs—which are just as capable Wait, am I misunderstanding you? I feel like I've had a head injury or something, because I've never heard of an open source LLM that's as capable as GPT-4 (in most scenarios).

Only on specific domains, these models don't become generalists like GPT-4, they can become task experts for a single task.

Fine-tuned MedPalm is worse than GPT-4 on most Medical Challenge Tests. Fine-tuned Minerva is much worse on arithmetic benchmarks.

The LLM space is just different. There's no guarantee a fine-tuned model will beat a bigger generalist one.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#160

Post by famouswaffles »

albystein wrote: post #46
gojomo wrote: post #34

Not sure I understand the excerpt from the referenced paper. Is it saying that part of its more-efficient inferencing relies on mixing tokens from completely-separate inputs – eg, from other users? And then, depending on what other inputs chance into the same grouping, the relative assignment-to-'experts' varies, and thus the eventual completions? If so, I'd see that as not just introducing non-determinism, but also…

this seems like a plausible outcome, and if true could spell disaster for OpenAI models relative to the competition and open source models. Currently, reliability is one of the core obstacles preventing widespread adoption of LLMs in many business critical workflows. And if these rumors, that GPT-4 is inherently un-deterministic and unreliable, are true then most enterprises are better off finetuning open source LLMs…

Fine-tuned MedPalm is worse than GPT-4 on most Medical Challenge Tests. Fine-tuned Minerva is much worse on arithmetic benchmarks.

The LLM space is just different. There's no guarantee a fine-tuned model will beat a bigger generalist one.

Post reply on HN