Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

81–90 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#81

Post by ares623 »

Jensson wrote: post #68
atoav wrote: post #53

Earlier quoted context omitted.

Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

> Now if we phrase the same question slightly differently we would expect to get a slightly different answer. You wouldn't get this from an LLM though, a tiny change in starting point gets a massive change in output, its a chaotic system.

Maybe predictability is what is meant?

Re: Defeating Nondeterminism in LLM Inference

#82

Post by ako »

themeiguoren wrote: post #80

A bit off topic from the technical discussion but does anyone recognize what blog layout or engine this is? I really like the layout with sidenotes and navigation.

Seems like a Thufte inspired style, something like this: https://clayh53.github.io/tufte-jekyll/articles/20/tufte-sty...

Re: Defeating Nondeterminism in LLM Inference

#83

Post by tonyhart7 »

skybrian wrote: post #62
kazinator wrote: post #38

Earlier quoted context omitted.

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?

oh so you want it to be thinking???? now we talking

Re: Defeating Nondeterminism in LLM Inference

#84

Post by Zacharias030 »

kazinator wrote: post #38
lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

While I get that this is how LLMs work, I think you should think backwards from the user / from what AI as a field is aiming for and recognize that the „naive“ way of the parent to ask for reliable responses no matter what the „context“ is, is exactly what a good AI system should offer.

„The context is the input“ betrays a misunderstanding of what (artificial) intelligence systems are aiming for.

Re: Defeating Nondeterminism in LLM Inference

#85

Post by sudohalt »

lairv wrote: post #57
sudohalt wrote: post #37

cool project but if this is what you are producing with $2 billion funding, i doubt you will survive. This is the type of article a grad student would write over a weekend.

on the contrary this makes me bullish about their team, it shows that people here care about the craft

The team is good, and I enjoyed the read. But this is just an engineering blog post. They're promoting this like it's ground breaking research and it's on their front-page. Ultimately this paper is not very meaningful and just a fun debugging session.

I've seen this play out dozens of times. So many startups that have come and go in the bay area were composed of extremely talented individuals, but almost all of them failed.

Re: Defeating Nondeterminism in LLM Inference

#87

Post by sudohalt »

sailingparrot wrote: post #24
nowittyusername wrote: post #21

I am baffled that I still run against these statement years after LLM's have been around. LLM's are deterministic and always have been. The reason people are having issues with them is because they are basing their assumptions on api based experiments. Like my man, how can you be making these statements when you haven't done the due diligence of running the LLM on your own hardware with all of the variables locked do…

I suggest you look up the name of the main author of TFA before assuming they don’t know what they are talking about. This is literally one of the most knowledgeable person on the topic. I think you are the one that hasn’t peeled enough layers to connect with what they are saying.

1. they aren't, they are just popular online. 2. the author has nothing to do with the original comment. Why do you think academic reviews are double blind?

Re: Defeating Nondeterminism in LLM Inference

#88

Post by emharsha1812 »

I think this is an excellent article which addresses the issue that I personally have been thinking about a long time. And no its not just some slop they put but actual an engineering blog(with open source code and reproducible results!) I think the company is off to a good start

Re: Defeating Nondeterminism in LLM Inference

#89

Post by riazrizvi »

raincole wrote: post #78
riazrizvi wrote: post #56

Earlier quoted context omitted.

Me: What’s an example of a dice roll? LLM: 1 “Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.

> LLM: 1 A perfectly acceptable answer. If it answers 1 every time it's still a perfectly acceptable answer.

So is ‘2’ or ‘3’ or ‘19’ or ‘99’ or ‘a jam sponge cake with gaming dice for frosting’… The point is in natural language there are many perfectly acceptable answers. Usually any particular answer is arbitrary, and it would probably be undesirable to have the same answer everytime. For a majority of use cases.

Re: Defeating Nondeterminism in LLM Inference

#90

Post by reasonableklout »

Some great discussion on twitter: https://x.com/thinkymachines/status/1965826369721623001

Seems a buried lede is that on-policy RL is unlocked by bitwise identical results between training and sampling. I'm not an expert here but my understanding is that this would allow for stronger guarantees about deployment/training alignment for the RL training that the labs already do.

I don't fully understand the BigMath example though. They show that off-policy RLVR requires off-policy correction, which avoids divergence, but is suboptimal because it results in noisy rewards. Then they say "we fixed the sampler and trainer numerical mismatch, which allows for on-policy RL, look how much better it is." It's not clear to me whether this is an artificial example that deliberately uses different trainer/sampler setups, or if it's actually impossible to have the same numerics between trainer/sampler without their fixes (even if we use same batch size, no atomics, etc.).

Post reply on HN