Obligatory "The Conspiracy Against High Temperature Sampling": https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
where can one learn what top_k and top_p mean?
Gemini last models: temperature, top_p, and top_k are deprecated and ignored
11–20 of 53 posts
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#12> To improve determinism, define a system instruction with explicit rules for your specific use case. Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.
It is guaranteed to work worse than top_k=1, that’s for sure.
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#13> To improve determinism, define a system instruction with explicit rules for your specific use case. Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.
[flagged]
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#14Obligatory "The Conspiracy Against High Temperature Sampling": https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
where can one learn what top_k and top_p mean?
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#15Obligatory "The Conspiracy Against High Temperature Sampling": https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
where can one learn what top_k and top_p mean?
ironically, any frontier LLM will easily generate a tutorial at any detail you like explaining what these are.
if don't have time for that, just know that these are technical parameters that affect how likely it is an llm will produce the same result after being asked the same question.
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#16thank god, these parameters are so confusing
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#17> To improve determinism, define a system instruction with explicit rules for your specific use case. Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.
It is guaranteed to work worse than top_k=1, that’s for sure.
Depends on what you mean by determinism.
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#18Possible reasons:
- They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.
- They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".
[1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of...
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#19Obligatory "The Conspiracy Against High Temperature Sampling": https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
where can one learn what top_k and top_p mean?
Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored
#20Along with everything else. These parameters can make speculative decoding less accurate increasing the inference cost.