"Last" or "latest"? Those are rather different.
Gemini last models: temperature, top_p, and top_k are deprecated and ignored
21–30 of 53 posts
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#22Earlier quoted context omitted.
where can one learn what top_k and top_p mean?
ironically, any frontier LLM will easily generate a tutorial at any detail you like explaining what these are. if don't have time for that, just know that these are technical parameters that affect how likely it is an llm will produce the same result after being asked the same question.
Is an llm able to explain to itself what these parameters do, and change its own parameter settings?
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#23I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#24Obligatory "The Conspiracy Against High Temperature Sampling": https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
where can one learn what top_k and top_p mean?
The posted answers are either behind a paywall or very obtuse so I'll just explain. I'll assume you know what tokens are.
A models output is not a single token, but a list with the probability for all the tokens that it knows, so we need to use a sampler to select the token that it's going to be the next token in the sentence. For example a simple greedy sampler will choose the token with the highest probability, but samplers normally pick a random token weighted by probability. A model usually knows about ~250 thousand tokens and the probability of some of these tokens are gonna be high, but the vast majority is close to but not actually 0% so there's a chance the sampler might pick some random token that doesn't make much sense, so we filter tokens.
top_k filters the tokens so that only the k top tokens are selected. So top_k=50 will filter those 250k tokens to only 50. This is assuming the list of tokens is sorted by probability.
top_p filters the top tokens until a percentage is accumulated. So if for example if you set the top_p to 0.6 and the model gave the top token a 0.5 (50%) probability and the second top token a 0.2, those 2 token accumulated to 0.7 which is greater than what you set it to (0.6) so no more tokens are selected. If this ran after top_k=50 it'll turn the list of 50 tokens into one of 2.
After each filter parameter is processed, the probability of the tokens is adjusted to sum to 1 (100%), Also note that order of operation here matters, i.e. top_p could be applied before top_k, but most providers follow what's on huggingface, I think I've only seen different implementation in certain local model hosting frameworks.
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#25"Last" or "latest"? Those are rather different.
hehe...
fwiw, the un-editorialized title is "Using the *latest" Gemini models"
I speculate OP wanted to put focus on their chosen detail in the title...
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#26> To improve determinism, define a system instruction with explicit rules for your specific use case.
"Please be deterministic".
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#27[flagged]
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#28> To improve determinism, define a system instruction with explicit rules for your specific use case.
What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one.
The latest OAI models have done the same thing. I'm currently adding random variation to prompts to compensate for the lack of higher temperature sampling.
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#29Obligatory "The Conspiracy Against High Temperature Sampling": https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
wow I wish I knew this post before I installed various UIs with tons of temp/top-X sliders.
They are extremely confusing.
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#30Earlier quoted context omitted.
ironically, any frontier LLM will easily generate a tutorial at any detail you like explaining what these are. if don't have time for that, just know that these are technical parameters that affect how likely it is an llm will produce the same result after being asked the same question.
Is an llm able to explain to itself what these parameters do, and change its own parameter settings?
An LLM is stateless so every time you send a message a command is run with the parameters, the prompts, and a copy of the chat history when using one of those chat interfaces, and then it returns text. If the output includes explaining it to an LLM that would then be included in the copy of the chat history placed in the new command.
It would be a bit like opening a text editor and typing in that you want to increase the font size. Someone external to the text would have to come along and click the font options.