71–80 of 211 posts
Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number. But instead if I ask it to generate 100 samples, it actually works pretty well. "You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Generate 100…
> Write a program for a weighted random choice generator. Use that program to say ‘left’ about 80% of the time and 'right' about 20% of the time. Simply reply with left or right based on the output of your program. Do not say anything else. Running once, GPT-4 produced 'left' using: import random def weighted_random_choice(): choices = ["left", "right"] weights = [80, 20] return random.choices(choices, weights)[0] #…
> You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else. Give me 100 of these random choices in a row.
It generated the code behind the scenes and gave me the output. It also gave a little terminal icon I could click at the end to see the code it used:
import numpy as np
# Setting up choices and their weights
choices = ['left', 'right']
weights = [0.8, 0.2]
# Generating 100 random choices based on the specified weights
random_choices = np.random.choice(choices, 100, p=weights)
random_choicesSometimes when you ask chatgpt 4 for a random number it... writes python code to choose a random number, runs it, then tells you the response: https://chat.openai.com/share/a72c2d8c-c44e-4c89-b6bc-b0673c... One way of doing it, I suppose.
Technically speaking, it's the right way to do it.
"You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" Humans would say "Left" 100% of the time in a zero-shot scenario as well. Intuitively, your first response is going to be "left" since it has the 80% probability. You'd balance your answers over time when you realized you were closer to…
> Humans would say "Left" 100% of the time in a zero-shot scenario as well. How can you know what all humans would do? If the humans interpreted the task correctly, that is, if they understood they will only be asked once, but in a hypothetical repeated experiment the result should still be 80/20, they would certainly not always say "left".
Because you're really asking what they think the first response would be. That's left. If I knew a machine would pick left 80% of the time, I would bet left 100% of the time. And I'd be right about 80% of the time, which isn't perfect, but is profitable.
I wonder if you could actually fine tune an LLM to do better on this. As some of the comments point out, the issue here is that the possible output probabilities combined with the model temperature don't actually result in the probabilities requested in the prompt. If you trained on specific generated data with real distributions would it learn to compensate appropriately? Would that carry over to novel probability p…
If the temperature was not zero, then it seems technically possible for the output tokens to weighted closely enough in probability to each other in a way such that the randomization from temperature causes tokens to be printed in the appropriate distribution.
However, I'm not an LLM expert, but I don't think that people use a "temperature" while training the model. Thus the training step would not be able to learn how to output tokens in the given distribution with a given temperature because the training step does not have access to the temperature the user is using.
EDIT: I made the assumption that the LLM was not asked for a sequence of random numbers, but only one number per prompt. I think this fits the use case described in the article, but another use case might be asking for a sequence of such numbers, in which case training might work.
Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number. But instead if I ask it to generate 100 samples, it actually works pretty well. "You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Generate 100…
It actually is surprising, and you should be surprised rather than post hoc justifying it, because the logits should reflect the true random probability and be calibrated in order to minimize the prediction loss. Putting ~100% weights on 'heads' is a terrible prediction!
And the LLM logits are in fact calibrated... before they go through RLHF and RLHF-derived dataset training. (Note that all of the models OP lists are either non-base tuned models like ChatGPT, or trained on data from such models, like Phi.) This was observed qualitatively when the 3.5 models were first released to the Playground, documented by the GPT-4 paper, and the 'flattened logits' phenomenon has been found many times since, not just by OP, and mostly by people totally ignorant of this phenomenon (despite being quite well known).
This is just one of those things, like BPE-related errors, that we're doomed to point out again and again in the Eternal September of LLMs.
"You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" Humans would say "Left" 100% of the time in a zero-shot scenario as well. Intuitively, your first response is going to be "left" since it has the 80% probability. You'd balance your answers over time when you realized you were closer to…
> Humans would say "Left" 100% of the time in a zero-shot scenario as well. How can you know what all humans would do? If the humans interpreted the task correctly, that is, if they understood they will only be asked once, but in a hypothetical repeated experiment the result should still be 80/20, they would certainly not always say "left".
I don't know if our decision making processes are deterministic or quantum-random. If the former, then if you could reset a human mind and ask the same question, you would necessarily always get the same answer, whatever that happened to be.
Interesting.
"You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" Humans would say "Left" 100% of the time in a zero-shot scenario as well. Intuitively, your first response is going to be "left" since it has the 80% probability. You'd balance your answers over time when you realized you were closer to…
They do not! And you should not just make up assertions like these. You don't know what humans would say. In fact, in polls, they wind up remarkably calibrated. (This is also covered in the cognitive bias literature under 'probability matching'.) People do this poll on Twitter all the time.