451–460 of 461 posts
Earlier quoted context omitted.
Training impossibilities aside, how would you even make an optimization loop for "fun"?
Brain-computer interface, maybe. Hook a million play-testers up to the output and have it iterate. ..feels like there was a black mirror episode about that though
Earlier quoted context omitted.
Agree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting. A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and firs…
Have you seen pelicans in Simon Willison’s tests? It is still not a pelican on bike I would like to publish :) Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time. It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see…
Earlier quoted context omitted.
> We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". First sentence of the post is clear that it's not solved.
Might be my reading comprehension, but that reads to me like it's implying we're leaving the territory because it's solved and now need something harder.
When I start to leave my house in the morning, I have not yet gone through the door, but I'm on my way. I walk towards the door to leave the house. I'm leaving the house as I pass through the door. I look back and can see that I've left the house.
Earlier quoted context omitted.
Taking a single paragraph of literary text, which is abstract and ambiguous, and converting it into a 3D animation requires an enormous amount of implicit knowledge about spatial relationships, intuitive physics, everyday objects, and so forth. Not to mention the mathematics of 3D transformations and computer graphics more generally. Saying that it's indicative of no more than three.js coding ability is absurd.
Why does it require knowledge about spatial relationships?
It’s something that specifically stopped happening in the smarter models.
Same way the “pelican test” became worthless beyond a certain point and just some idea of “I like this one better” not technically more competent in any way.
Earlier quoted context omitted.
Might be my reading comprehension, but that reads to me like it's implying we're leaving the territory because it's solved and now need something harder.
"Starting to leave" is not the same as "leave" or "left". When I start to leave my house in the morning, I have not yet gone through the door, but I'm on my way. I walk towards the door to leave the house. I'm leaving the house as I pass through the door. I look back and can see that I've left the house.
Anyways, if it's just semantic misunderstanding, I have no issue. That means the fact is, models still can't draw a pelican riding a bike? With maybe the exception of Fable 5?
I don't think it's a bad way to benchmark new models, I just find it concerning that the author implies that "pelican on a bicycle" has been exhausted. At the risk of making overly broad, unfalsifiable claims I think multi-year exposure to AI content has dramatically raised our expectations for speed and volume but lowered them for quality. We see a very janky pelican and declare the problem solved.
I worked with an LLM to build a ~3D animation of the Back to the Future delorean Time Machine as a way to spice up the hero on a docs page. That took a fair amount of custom tuning and I had to create a tuning view to get some of the behaviors right. But it was enough fun that I generalized it to take in ~any scene description from a film. It goes out and gets more detailed descriptions and film stills if available b…
A simple prompt that still stumps frontier LLMs most of the time is “create a pinball game”. They’ll put all the right pieces there but then fail to arrange them such that the game is truly playable. They’ll put a wall in the way of the launch chute so the ball can’t be launched. Or the flippers will pivot the wrong way. Or there will be holes such that the ball drops off the bottom without getting within reach of th…
We don't use these models by way of "one shot" so I don't see why it's relevant. It's clearly useful to let them iterate.
Earlier quoted context omitted.
> touching the limit of what one can reasonably expect If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that. And Elon is already propagating the age of abundance where money won't exist anymore, right before calling the interviewing journalist dishonest and deservedly losing public trust. Smh my head.
> If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that. What knowledge workers do you know that have excellent drawing skills? I worked in a design agency and for a couple of years, each week me and a few other people would attempt to sketch a member of our group: one person would be the model and sit still, and everyone els…