81–90 of 123 posts
Earlier quoted context omitted.
Should we also try to teach children geometry and test them on calculus?
For centuries we only taught children geometry and one of them invented calculus.
Earlier quoted context omitted.
Quite crappy? It seems to me like the current SOTA is working plenty good enough for most use cases and like the highest impact (practical) way to improve right now is going to be advancements in domain knowledge acquisition and retention.
I sure haven't seen any of those "plenty good" results yet - current SOTA seems to be about as useful as semi-coherently gluing together random Google results. Good enough to perhaps replace a minimum-wage worker, not good enough to provide actual value when you care about the quality of the result. This paper seems to suggest that significant advancement in domain knowledge acquisition and retention is exactly the p…
Replacing CLIP with an LLM is the current meta in image generation models, specifically because of that lack of generalisation. This isn’t a surprise to anyone.
Earlier quoted context omitted.
How would the model know what zebra was if it had never seen it? Same is true for humans.
When I was little, zebras were described to me as black and white stripped horses. Without even seeing one, I'm sure anything who has seen a horse could then merge those two concepts to create a close to accurate picture of what a zebra is. If AI is supposed to resemble a human mind with ability to learn, then it must be able to learn from a blanker slate. You don't teach the human before it is born, and in this comp…
Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.
Earlier quoted context omitted.
How would the model know what zebra was if it had never seen it? Same is true for humans.
When I was little, zebras were described to me as black and white stripped horses. Without even seeing one, I'm sure anything who has seen a horse could then merge those two concepts to create a close to accurate picture of what a zebra is. If AI is supposed to resemble a human mind with ability to learn, then it must be able to learn from a blanker slate. You don't teach the human before it is born, and in this comp…
https://britishlibrary.typepad.co.uk/digitisedmanuscripts/20...
Earlier quoted context omitted.
RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?
There's tons of optimization work left, but by it's nature optimization tends to push the limits of what we currently have, and rarely allows for substantial improvements. There are some major limitations to LLMs that aren't going to be "optimized" away. At the end of the day LLMs are Monte Carlo samplers over a latent, compressed representation existing human text data. Many of the tasks people hope LLMs will achiev…
ie., P(mat|the cat sat on the) is a distribution over, say, 100k words. Whereas, P(the cat sat|on the mat) is a distribution over 100k^3 words.
Part of the illusion of an LLM is that we produce create text in such a highly regular way that a mere distribution over 100k iterated, say 500 times, gives you a page of text as if modelling 100k^500.
Earlier quoted context omitted.
Should we also try to teach children geometry and test them on calculus?
This illustrates two ways of teaching I’ve experienced both, each at a different university In one, professors would teach one thing then ask very different (and much harder) questions on tests In the other, tests were more of a recap of the material up to that point I definitely learned a lot more in the second case and was a lot more motivated. It also required more effort from the professors The two methods also t…
I think you are misunderstanding the experience.
The first (harder questions) is testing your understanding of the material and problem. Can you applying the material to solve a novel problem? Do you understand the material not just the mechanics. Do you understand how it would interelate it with other problems? Do you understand the limitations?
The second is just regurgitation. This is great for rote skills, but this isn't really learning. This is grinding until you can reproduce. These are the kinds of skills that are easily automated. This is not what we should be testing our kids.
Earlier quoted context omitted.
Shouldn't there be enough known-good training content that we can use it to determine if a document is worth including in a training set?
There's not enough human workers to validate that at scale. If you want ML to do it... well that's a bit of a catch-22. How would an ML algorithm know if data is good enough to be trained on unless it has already been trained on that data?
This seems doable, and, I think something like it is already done?
Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.
But I do feel confident that an AI winter is not on the horizon solely due to the overhead of implementation that we currently have. Just with currently existing AI, it would take years for the economy to fully leverage the abilities that are available. I'm confident we have transformative AI. So it won't feel like a winter for several years as we actually succeed in optimizing, implementing, and productizing current technology in other industries.
I've always been rather upset that it's fairly common to train on things like LAION or COCO and then "zero shot" test on ImageNet. Zero shot doesn't mean a held out set, it means disjoint classes. You can't train on all the animals in the zoo with sentences and then be surprised your model knows zebras. You need to train on horses and test on zebras.
How would the model know what zebra was if it had never seen it? Same is true for humans.
It depends on the zero-shot experiment. Let's look at two simple examples
Example 1:
We train a classifier that classifies several animals (and maybe other things). For example, you can use the classic CIFAR-10 dataset which has labels: airplane, automobile, bird, cat, deer, dog, frog, __horse__, ship, truck. The reason I underlined horse is because you want your model to classify the zebras as horses!
The reason this is useful is for measuring the ability to generalize. At least in our human thinking framework we'd place a zebra in that bin because it is the most similar (and deer should be the most common "error"). This can help us understand the network and we'll be pretty certain that the network is learning the key concepts of a horse when trying to classify horses rather than things like textures, colors, or background elements. If it frequently picks ships your network is probably focusing on textures (IIRC CIFAR has ships with the Dazzle Camo[0] and that's why I threw "ship" out there).
Example 2:
Let's say we train our network on __text__. In this case it can get any description of a zebra that it wants. In fact, you'd probably want to have a description of what it looks like!
The what we might do is take that trained text network, and attach it to a vision classifier. For simplicity, let's say that was trained on CIFAR-10 again. We then tune our LM + CV model so that it can match the labels of CIFAR-10 (basically you're tuning to ensure the networks build a communication path, otherwise it won't work). Here we end up testing our model's actual understanding of the zebra concept. It again should pick horse as the likely class because you've presumably had in the training text some description that compares zebras to horses.
-----
So really the framework of zero-shot (and few-shot) is a bit different. We're actually more concerned about clustering and you should treat them more similar to clustering algorithms. n-shot frameworks really come from the subfield of metalearning (focusing on learning how networks learn). But as you can imagine, these concepts are pretty abstract, but hey, so are humans (that's why we see a log as a chair and will situationally classify it as such, but let's save the discussion of embodiment for another time).
In either example I think you can probably see how a toddler could do similar tasks. You can ask which of those things the zebra is most similar to and you'd be testing the toddler's visual reasoning. The text one might need be a little older but it could be a great way to test a child's reading comprehension. Does this make sense? Of course machines are different and we need to be careful with these analyses (which is why I rage against just comparing scores/benchmarks, these mean very little), because the machines may be seeing and interpreting things differently than us. So really the desired outcome depends on if you're testing for what the machine knows/understands (you need to do way more than what we discussed above) or if you are training a machine to think more similar to a human (then we can rely pretty close to exactly what we discussed).
Hope this makes more sense.