21–30 of 123 posts
My biggest worry is the idea of generating data as training data. We're obviously already unwittingly doing this, but once someone decides to augment low-volume segments of the dataset with generative input, we're going to start getting some really crappy feedback loops.
Well, you’re learning from data you generate.
My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids. That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.
This deserves to be on the front page. The authors ask whether image-to-text and text-to-image models (like CLIP and Stable Diffusion) are truly capable of zero-shot generalization. To answer the question, the authors compile a list of 4000+ concepts (see paper for details on how they compile the list of concepts), and test how well 34 different models classify or generate those concepts at different scales of pretra…
My biggest worry is the idea of generating data as training data. We're obviously already unwittingly doing this, but once someone decides to augment low-volume segments of the dataset with generative input, we're going to start getting some really crappy feedback loops.
Do you ever have novel ideas while walking or in the shower? Well, you’re learning from data you generate.
Sure. I'm producing human output from human input in a generally unconstrained, limitless way.
This is producing approximated human output from approximated human input. That second level of abstraction will be constrained, ultimately, by the limits of input.
Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.
We're just scratching the surface of what's possible with the current state of the art. Even if there are no major advances or breakthroughs in the near future, LLMs and associated technologies are already useful in many use cases. Or close enough to useful where engineering rather than science will be sufficient to overcome many (though not all) of the shortcomings of current AI models. Never mind grandiose claims about AGI, there's enough utility to be gotten out of the limited LLMs that we have today to keep engineers and entrepreneurs busy for years to come.
This feels like the worst possible outcome of the current AI hype. We've essentially been ripping off the entire internet and feeding it to the models already, spending many billions of dollars in the process. It's pretty much the largest possible dataset you can currently get, and due to the ever-increasing and now rapidly accelerated AI poisoning of the internet most likely the largest possible dataset which will e…
15 months ago, general-purpose LLMs that have not been specifically trained on legal reasoning could score better than 90% of humans on the multistate bar exam, and these are humans who actually completed law school.
General-purpose LLMs get similar results in medicine, and when the models are fine-tuned for medical diagnosis they're even better.
And that was more than a year ago. Those who have seen current models not yet released tell us they'll make the current state of the art look like toys.
Progress is still tracking the steep part of the S-curve, and there's no indication that they're near the top yet.
Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.
Again?
My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids. That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.
Every exponential is really an s curve?