Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-determinism in GPT-4 is caused by Sparse MoE

152334h.github.io

Post reply on HN
Print view View original

21–30 of 186 posts

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#21

Post by nerdponx »

dylan604 wrote: post #17
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

I want to know what a non-boring flight would be like

High turbulence definitely makes it less boring. So will a crying baby, disruptive passenger, or someone getting sick. After a few of those, you'll prefer the boring flights.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#22

Post by 152334H »

dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

just set up a desktop service to randomly open a paper once every few hours

if they're not too boring, and you're not doing anything important, you'll read it for fun

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#23

Post by jldugger »

jldugger wrote: post #19
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

For just getting started I recommend collections: 1. Ideas That Created The Future[1]. It's a collection of fiftyish classic CS papers, with some commentary. 2. Wikipedia's list[2]. 3. Test of Time awards[3]. These are papers that have been around for a while and people still think are important. 4. Best paper awards[4]. Less useful than ToT as not every best paper is actually that good or important, and sometimes th…

From there, just keep a reading queue. If you notice a particular journal is a good source of material, consider subscribing to it.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#24

Post by 152334H »

dylan604 wrote: post #17
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

I want to know what a non-boring flight would be like

https://www.youtube.com/watch?v=iFImKMjM-q4

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#25

Post by LordShredda »

dylan604 wrote: post #17
dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

I want to know what a non-boring flight would be like

Snakes on a plane

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#26

Post by refulgentis »

cubefox wrote: post #13
152334H wrote: post #8

Earlier quoted context omitted.

Thanks. I'm really no expert (:P) on MoE research; I just noticed what was written in the Soft MoE paper and felt a need to check. The non-deterministic outputs are really similar, yeah, if you check the gist examples I linked https://gist.github.com/152334H/047827ad3740627f4d37826c867a... . This part is at least no surprise, since the randomness should be bounded. I suspect OpenAI will figure out some way to reduce…

I don't think this commitment had any plausibility. Token "probabilities" only have a straightforward probabilistic interpretation for base models. In fine-tuned models, they do no longer represent the probability of the next token given the prompt, but rather how well the next token fulfills the ... tendencies induced by SL and RL tuning. Which is presumably pretty useless information. OpenAI has no intention to pro…

Topic laundering, the probabilities are the probabilities, you don't suddenly get wrong probabilities with more training on more data

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#27

Post by lucubratory »

FanaHOVA wrote: post #10
refulgentis wrote: post #4

This is _excellent_ work, I've been adamantly against MoE for a set of reasons, this is the first compelling evidence I've seen that hasn't been on Substack or a bare repeating of rumor. I had absolutely no idea GPT4 was nondeterministic and I use it about 2 hours a day. I can see why a cursory looking wasn't cutting it, they "feel" the same in your memory, a lot of similar vocab usage, but are formatted entirely dif…

> I've been adamantly against MoE for a set of reasons Such as?

It was completely unsubstantiated, based on rumours from a blog, but everyone repeated it as fact.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#29

Post by osmarks »

alpark3 wrote: post #12

_If_ 3.5 is a MoE model, doesn't that give a lot of hope to open source movements? Once a good open source MoE model comes out, maybe even some type of variation of the decoder models available(I don't know whether MoE models have to be trained from scratch), that implies a lot more can be done with a lot less.

It would be bad for single-consumer-GPU inference setups.

Re: Non-determinism in GPT-4 is caused by Sparse MoE

#30

Post by throwawayadvsec »

dudus wrote: post #14

Off topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.

Step 1. Find papers you're interested in Step 2. Open them Step 3. read them
Post reply on HN