Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini last models: temperature, top_p, and top_k are deprecated and ignored

ai.google.dev

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#41

Post by vintermann »

nodja wrote: post #24
NooneAtAll3 wrote: post #11

Earlier quoted context omitted.

where can one learn what top_k and top_p mean?

The posted answers are either behind a paywall or very obtuse so I'll just explain. I'll assume you know what tokens are. A models output is not a single token, but a list with the probability for all the tokens that it knows, so we need to use a sampler to select the token that it's going to be the next token in the sentence. For example a simple greedy sampler will choose the token with the highest probability, but…

I used to try to train models ages ago (that is, a couple of years), and I never found out how they got good results from top_k in the first place. My problem was that as soon as the model had generated an unlikely token, it was quickly steering towards spaces it had never seen during its training. It found itself in unmapped territory, where anything might happen. With high temperatures, it would then generate noisy garbage, with low temperature it would generate repetitive garbage. We tried to fight this with rollout strategies and backtracking, i.e. if you find yourself in a space where all tokens seem equally likely, you're probably in junk land.

Our tricks didn't work very well. But I didn't manage to keep up well enough to learn what worked.

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#42

Post by orbital-decay »

bob1029 wrote: post #28

> To improve determinism, define a system instruction with explicit rules for your specific use case. What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one. The latest OAI models have done the same thing. I'm currently adding random var…

Current post-trained models are stochastically and semantically collapsed to a large extent, you'll never get meaningful semantic diversity if your prompt stays the same. This makes the temperature brittle, the model becomes dumb much earlier than meaningfully diverse. There are fancy samplers that claim to overcome this (comments ITT mentioned them), but a) you need a local inference stack to use them and b) I've yet to see any definitive evidence they add any deep semantic diversity without dumbing the model down, most of what I've seen is pretty superficial, including distribution-aware samplers.

>I'm currently adding random variation to prompts

This is better than naive temperature tweaks, if done right.

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#43

Post by henriquecm8 »

david-gpu wrote: post #37
nichohel wrote: post #21

"Last" or "latest"? Those are rather different.

The OP does not appear to be a native English speaker. And, for example, languages like Spanish don't clearly differentiate between "last" and "latest".

I am not a Spanish speaker, but Portuguese is usually similar. In it we say "último" for last, and "mais recente" for latest, so while there isn't a single word translating, we still have its equivalent. I imagine it's the same for Spanish.

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#44

Post by StrauXX »

herpderperator wrote: post #23

I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?

I'd expect that sampling would happen in software. Probably the hardware system would output a vector of probabilities over the tokenspace, just as the nets do when run in software.

I don't know though and am not aware of any docs going into detail here. That being said, sampling is really cheap. So implementing it in hardware wouldn't be worth it.

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#45

Post by marcinjachymiak »

It's not surprising since these are all reasoning models now. Token sampling/decoding isn't exposed because the reasoning models are trained around specific configurations. That's why it's been replaced with "reasoning effort" instead (low, medium, high "thinking" levels).

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#46

Post by kittikitti »

The API endpoints are less useful if you can't even fine-tune these parameters. These endpoints used to be developer oriented and this sort of change would be a huge cause of concern. If you primarily vibe-code, this is not a problem but understand that they're making their product worse on purpose.

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#47

Post by gregdaniels421 »

dev_dan_2 wrote: post #40
franky47 wrote: post #26

> To improve determinism, define a system instruction with explicit rules for your specific use case. "Please be deterministic".

> improve determinism Oh well. I might be to picky here, but how I see things, determinism cannot be improved or worsened, but achieved or not achieved. Or Partially archieved, when analyzing a system that has both components that are deterministic or non deterministic. There are times to think in absolutes, and when talking about deterministic behavior of technical systems, this is one of them. Join the sith side, w…

Probably what is meant is consistency instead of determinism

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#48

Post by greatgib »

froh wrote: post #25
nichohel wrote: post #21

"Last" or "latest"? Those are rather different.

hehe... fwiw, the un-editorialized title is "Using the *latest" Gemini models" I speculate OP wanted to put focus on their chosen detail in the title...

Indeed, the focus was on this section but it is direct link to the relevant paragraph.

For the "last/latest", it is the result of trying to trim the title enough to enter in the size limited title box and then I didn't notice the typo.

At first I wanted to put the 3.5/3.6 model number because "latest" is not clear and time dependant but it was too long then.

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#49

Post by greatgib »

kittikitti wrote: post #46

The API endpoints are less useful if you can't even fine-tune these parameters. These endpoints used to be developer oriented and this sort of change would be a huge cause of concern. If you primarily vibe-code, this is not a problem but understand that they're making their product worse on purpose.

Same thing as the new APIs like assistant API that rip us of the control of the history of messages for the context that is now hidden server side.

Re: Gemini last models: temperature, top_p, and top_k are deprecated and ignored

#50

Post by david-gpu »

henriquecm8 wrote: post #43
david-gpu wrote: post #37

Earlier quoted context omitted.

The OP does not appear to be a native English speaker. And, for example, languages like Spanish don't clearly differentiate between "last" and "latest".

I am not a Spanish speaker, but Portuguese is usually similar. In it we say "último" for last, and "mais recente" for latest, so while there isn't a single word translating, we still have its equivalent. I imagine it's the same for Spanish.

In Spanish, "último" is ambiguous, and can mean either last or most recent.

Funnily, in Galician there is a distinction between "último" (most recent) and "derradeiro" (final). I would have guessed that it would be the same in Portuguese, given how closely related the two languages are.

Post reply on HN