Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

51–60 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#51

Post by giveita »

kybernetikos wrote: post #49

For fun over the last few days, I've built a compressor / decompressor that uses the logits from an LLM, for each token in the input, then takes the ranks and exponential goolomb encodes them. Then you work in reverse to regenerate the original It took me ages to get the prediction for the second token after "hello" to match the same as the prediction for the second token when running the model on the string "hello w…

What's the Weissman score? Or more seriously :) did it perform well. Sounds like it should. If more and more text is AI slop it should do well.

I dont fully understand what you said but I guess higher probability logits are encoded with fewer bits. If your text is the LLM output then you may need a bit or two per token?

Re: Defeating Nondeterminism in LLM Inference

#52

Post by ricardobeat »

gond wrote: post #48

I am still irritated by the name of the company. What is the reasoning behind these schemes? The hope that bits of the properties of legendary companies will rub off onto the new venture? As if naming the next best venture PARC will inevitably create a breakthrough in networking just by the arrangement of four letters.

Are you talking about the “Thinking Machines” company that shut down in 1994? Took me some digging to figure it out, doesn’t seem well-known enough to be the reason - it’s just a nice (and relatively obvious) name.

Re: Defeating Nondeterminism in LLM Inference

#53

Post by atoav »

riazrizvi wrote: post #41

Natural language is ambiguous. It needs to be. I think the approach here of trying to figure out how to make circles into squares, and argue why circles should be squares, is misguided. Discussions of this type are going to eventually morph into better understanding of how to accept ambiguity and randomness in language, and further shape it with other larger sub-patterns beyond the little proto-grammars that the QKV…

Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow.

If I ask the same model the same question I should be able to deterministically get the same answer.

Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

Re: Defeating Nondeterminism in LLM Inference

#54

Post by gond »

ricardobeat wrote: post #52
gond wrote: post #48

I am still irritated by the name of the company. What is the reasoning behind these schemes? The hope that bits of the properties of legendary companies will rub off onto the new venture? As if naming the next best venture PARC will inevitably create a breakthrough in networking just by the arrangement of four letters.

Are you talking about the “Thinking Machines” company that shut down in 1994? Took me some digging to figure it out, doesn’t seem well-known enough to be the reason - it’s just a nice (and relatively obvious) name.

Yes. Danny Hillis’ Thinking Machines Corporation, an AI company which created its own massive parallel processing supercomputer hardware.

“We are building a machine that will be proud of us” was their corporate motto. And that was in 1983.

One of those Machines is on view at the Computer History Museum in Mountain View. Back then, they could be ordered in “Darth Vader Black”, no kidding here. You can also see a couple of them (the CM-5) as the stereotypical supercomputer in the original Jurassic Park.

More here: https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporatio...

Re: Defeating Nondeterminism in LLM Inference

#55

Post by kkylin »

gond wrote: post #54
ricardobeat wrote: post #52

Earlier quoted context omitted.

Are you talking about the “Thinking Machines” company that shut down in 1994? Took me some digging to figure it out, doesn’t seem well-known enough to be the reason - it’s just a nice (and relatively obvious) name.

Yes. Danny Hillis’ Thinking Machines Corporation, an AI company which created its own massive parallel processing supercomputer hardware. “We are building a machine that will be proud of us” was their corporate motto. And that was in 1983. One of those Machines is on view at the Computer History Museum in Mountain View. Back then, they could be ordered in “Darth Vader Black”, no kidding here. You can also see a coupl…

And in the original Jurassic Park! https://www.google.com/search?q=jurassic+park+cm-5

Re: Defeating Nondeterminism in LLM Inference

#56

Post by riazrizvi »

atoav wrote: post #53
riazrizvi wrote: post #41

Natural language is ambiguous. It needs to be. I think the approach here of trying to figure out how to make circles into squares, and argue why circles should be squares, is misguided. Discussions of this type are going to eventually morph into better understanding of how to accept ambiguity and randomness in language, and further shape it with other larger sub-patterns beyond the little proto-grammars that the QKV…

Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

Me: What’s an example of a dice roll?

LLM: 1

“Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.

Re: Defeating Nondeterminism in LLM Inference

#57

Post by lairv »

sudohalt wrote: post #37

cool project but if this is what you are producing with $2 billion funding, i doubt you will survive. This is the type of article a grad student would write over a weekend.

on the contrary this makes me bullish about their team, it shows that people here care about the craft

Re: Defeating Nondeterminism in LLM Inference

#58

Post by kkylin »

kkylin wrote: post #55
gond wrote: post #54

Earlier quoted context omitted.

Yes. Danny Hillis’ Thinking Machines Corporation, an AI company which created its own massive parallel processing supercomputer hardware. “We are building a machine that will be proud of us” was their corporate motto. And that was in 1983. One of those Machines is on view at the Computer History Museum in Mountain View. Back then, they could be ordered in “Darth Vader Black”, no kidding here. You can also see a coupl…

And in the original Jurassic Park! https://www.google.com/search?q=jurassic+park+cm-5

[addendum: posted this too quickly & didn't see it in the comment above. duh.]

Re: Defeating Nondeterminism in LLM Inference

#59

Post by random3 »

gond wrote: post #48

I am still irritated by the name of the company. What is the reasoning behind these schemes? The hope that bits of the properties of legendary companies will rub off onto the new venture? As if naming the next best venture PARC will inevitably create a breakthrough in networking just by the arrangement of four letters.

The thinking is free marketing and the same reason trademarks were invented

Re: Defeating Nondeterminism in LLM Inference

#60

Post by kybernetikos »

giveita wrote: post #51
kybernetikos wrote: post #49

For fun over the last few days, I've built a compressor / decompressor that uses the logits from an LLM, for each token in the input, then takes the ranks and exponential goolomb encodes them. Then you work in reverse to regenerate the original It took me ages to get the prediction for the second token after "hello" to match the same as the prediction for the second token when running the model on the string "hello w…

What's the Weissman score? Or more seriously :) did it perform well. Sounds like it should. If more and more text is AI slop it should do well. I dont fully understand what you said but I guess higher probability logits are encoded with fewer bits. If your text is the LLM output then you may need a bit or two per token?

I used exponential golomb coding, so the rank 0 logit is encoded with a single bit, ranks 1 and 2 are encoded with three bits, ranks 3-6 are encoded with 5 bits, etc.

In terms of performance, I've not done any serious testing, but e.g. the wikipedia article on volcanos compresses to about 20% using GPT2. I've seen other strings compress even further.

The big issue is that while encoding is not unreasonable, decoding any significant amount of data is incredibly slow, since I'm doing a model run for every token in the output. It's bad enough that the scheme is probably unworkable as it is. I'm thinking about changing my code so that it streams out the tokens as it decodes them, so you're not just left there waiting for ages.

Post reply on HN