Live data from Hacker News

Non-determinism in GPT-4 is caused by Sparse MoE

152334h.github.io

181–186 of 186 posts

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#181

Earlier quoted context omitted.

Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I probably averaged 20 a week back in March when open source AI was booming in the wake of Llama and on the heels of GPT-4.

> Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I'm guessing that you don't actually dive into each paper to 100% understand it? I find it takes me at least 10 hours of reading/looking things up per paper before I could consider that I fully understand it. But that would mean, if I want to do 4 pa…

> I'm guessing that you don't actually dive into each paper to 100% understand it

Depends on the paper's content but there's often sections that you don't need to 100% understand to get value. For example, in survey papers, there's typically a section that is basically "what queries we typed in at the library." I skip those and I think you can too =)

For practical papers, sometimes the evaluation can be skimmed. Author's benchmarks are usually designed to be the most favorable to the paper's novel approach, so I don't spend too much time thinking about them.

Similarly, Related Work sections can be skimmed. If you're well read in the field, you probably won't learn anything from it, and if you're entirely unread "its like X but different because Y" isn't helpful as you have no idea what X is, beyond the one dense sentence the paper just gave you.

> And how much of the content would you estimate gets retained and can be recalled when you wish?

If I really want to remember a paper, it goes into Anki flashcards. This is rare, personally. Usually only for tech I support in prod.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#182
post #149

Earlier quoted context omitted.

Your comment, along with cpgxiii and n2d4’s are all really good. I have a question: suppose training and inference of an LLM were made to be deterministic at the cost of performance. Would the cost be “everything will take twice as long” or would it be more like “inference will take a week and training will take a couple lifetimes”? If it’s the latter, then it seems disingenuous to call this a “bug.” It’s like saying…

> That said, I agree with n2d4 that it’s stupid to insult the authors. Talk is cheap and building is hard. If your code offers an expectation of determinism then it's sloppy to not distinguish where there isn't determinism. There's nothing difficult about writing a comment to the effect of "this function is non-deterministic. For deterministic results, use X". The code is sloppy if the developers didn't consider dete…

There are flags[1] for that indeed. It feels like half of the people commenting here don't know all that much about the topic they're commenting upon

1: https://pytorch.org/docs/stable/generated/torch.use_determin...

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#183

Earlier quoted context omitted.

How much I understand, and how long it takes to get there, depends on a lot on how well-read I already am into a field. I can read and fully understand an ML paper in an hour or so. But 6 months ago it took me a day to get through a couple of ML papers and I did not fully grok the mechanics of things like attention heads. I'm more read in material science, chemistry, pharmacology, and cognitive science. Computer scie…

That's quite a range. How do you manage the signal-to-noise ratio? Normally that requires significant familiarity with the field, or a very specific query in mind. For example I only read papers in medicine when I'm researching an actual medical issue that I or someone else is having.

I follow a lot of highly respected researchers and (research minded) operators in the fields I'm interested in. Very often they post about papers of interest on X/twitter or their personal blogs. I also follow a handful of science communicators on YouTube who do short summary videos of papers of interest (Two-Minute-Papers, Anton Petrov, Sabine Hossenfelder, to name a few).

Other times I notice a general trend (ex. increasing discussion of a new paradigm X, more startups raising to work on Y, or a large chunk of talks at an annual conference being variations of Z).

Then I ask the forementioned academics and operators in my circle what papers I should read to get a handle on XYZ and/or simply follow the citations.

Given the amount of followers a lot of these researchers, operators, and science communicators have, I do not think I'm remotely unique in my efforts.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#184
post #82

Earlier quoted context omitted.

As the article says, cuBLAS is deterministic, but other CUDA primitives (eg. some of those in cudNN) are not. Yes, the non-determinism is being introduced somewhere, but that is splitting hairs. The point is that the primitives that you work with on GPUs are non-deterministic by design. I mostly take issue with you phrasing it as a bug and using it to insult the authors.

How is that splitting hairs? > The point is that the primitives that you work with on GPUs are non-deterministic by design. This is just blatantly wrong. There are _some_ operations that can be non-deterministic in some scenarios but they are not necessary. GPUs are deterministic. If you ask them to add a million floats in order, you get the same result every time. If you ask them to add a million floats in some arbi…

It's hard to call it a bug given that any concurrent float sum or product will be different in regards to changing the amount of concurrency. Even if you order the final value per thread before reducing the result will differ if you use a different amount of threads to split the problem.

Because in floating point arithmetic 1 + 2 + 3 + 4 is different than (1+2) + (3+4).

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#185
post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

[dead]

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#186

> It’s well-known at this point that GPT-4/GPT-3.5-turbo is non-deterministic, even at temperature=0.0 Interestingly, on another discussion there was a claim that setting the temperature to 0.0 made gpt-4 deterministic: https://news.ycombinator.com/item?id=36503146

This guy probably never did anything nontrivial with the API - you notice almost instantly that the chat models (both 3.5 and 4) are nondeterministic at 0 temperature. Source - built a documentation search bot and had it crap out on me on copy pasted prompts when I was demoing it.

If you want to make it deterministic, just cache the responses keyed by queries.
Post reply on HN