91–100 of 207 posts
There's nothing to infer with any greater accuracy than the unknown amount of noise in each sample. So, all inference is of unknown accuracy. Also known as useless.
That would give it some “learning” and I’d be curious if
1. Would it converge to a consistent shape at all? Or just bounce around random shapes day to day
2. Would it produce unicorns more often than 1/118 times?
The hardest part would be getting it to interpret its svg output without seeing it rendered. The multimodal model getting a rendered image would probably be much better, but maybe not!
It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…
I'm not sure what you mean by random sampling. If I sample a random SVG, I wouldn't expect to look like anything, let alone roughly like a unicorn.
It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…
Agreed. But have you seen the original talk? I believe he's trying to find an unicorn similar in style to the one generated by the original researcher. It's so sad that openai has a far more capable model internally that it can't give open access to because of safety (or any other argument).
Earlier quoted context omitted.
100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.
I believe the logic is fine. You seem to think you need multiple data points from the same version of the model (i.e. multiple samples per day at least) would be necessary to judge the actual performance on each particular day. That's worse logic. How would you visualize the very large sample you would get? Even with the current 118 samples (one per day) it's already difficult to find a pattern. Would you "average" t…
But maybe you could covert it to a linear monotonic measure?
You could pass it to an image recognition model and see record the degree to which it thinks it is an:
animal horse unicorn
Basically if it fails to be a unicorn, see if it is a horse and if it fails to be a horse check if it is an animal. This gives you some type of linear measure if you place these three measures along the same axis adjacently. Then you can transform each image to a floating point number and then characterize the distribution.
People keep forgetting that asking GPT to draw is like asking a human to imagine a 6D tesseract.
The prompt must be particularly bad. I managed to get a nicely looking unicorn at the first (and every subsequent) attempt.
The prompts are thus: > system: You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text. > user: Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks. What were yours?
It often gives svg files with incomplete paths, so i tweak the output to be a valid svg file.
I also enjoy the conversational description of the drawing.
Very often it's well described, ex. "this black circle is the head, and the grey element is the fog" and the drawing is crude like a child's drawing.
Landscapes often look better than animals, but animals are sometimes more entertaining.
Earlier quoted context omitted.
It takes a picture from GPT every day, so we'll be able to see if this was a fluke by looking at future days' outputs. I think it will work to track changes in the model.
Yes, this makes sense. You are agreeing with me. In order to see if it is just a fluke or not, you need a lot of samples so you can characterize the distribution yourself and try to see if it changed.
It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…
It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case stud…
Like most of HN.