711–720 of 860 posts
Earlier quoted context omitted.
I am suggesting that OpenAI's main product is "LLM that benchmarks the best." From that point, it is completely illogical not to train on at least some of the test data (or data that is very similar to the test data) so that you can fudge the numbers in your favor. You don't want to go too far, but overfitting a tiny bit will make you look like you have a significant edge. When someone says that your product isn't th…
Yeah and I'm saying I don't believe it. I don't know what you're talking about. GPT-4 is the best model out there by significant margin. That's coming from personal usage not benchmarks. A 10% drop in traffic the first month students are out of school is not "losing users quickly" lol. ChatGPT didn't gain public use waving benchmarks around. We didn't even know what they were until GPT-4's release. The vast majority…
Earlier quoted context omitted.
its not clear but can we also download the model with this Llama v2 Cog thing? EDIT: Meta is being extremely prompt, just got sent the download instructions https://twitter.com/swyx/status/1681351712718876673 also is it now Llama or LLaMA since the website says Llama? lol
Seems people here on HN were lucky. Signed up ~20 minutes ago and no download link, so surely they're doing some checking, it's not fully automated, unlike what others say here. On Huggingface, the following disclaimer has been put in place: > This is a form to enable access to Llama 2 on Hugging Face after you have been granted access from Meta. Please visit the [Meta website]( https://ai.meta.com/resources/models-a…
Earlier quoted context omitted.
It would be a bit of a scandal, and IMO too much hassle to sneak in. These models are trained on massive amounts of text - specifically anticipating which metrics people will care about and generating synthetic data just for them seems extra. But not an expert or OP!
You don't see an engineer at an extremely PR-conscious company at least checking how their model performs on popular benchmarks before rolling it out? And if its performance is lackluster, you do you really see them doing nothing about it? It probably doesn't make a huge difference anyway. I know those old vision models were overfitted to the standard image library benchmarks, but they were still very impressive.
Here are some benchmarks, excellent to see that an open model is approaching (and in some areas surpassing) GPT-3.5! AI2 Reasoning Challenge (25-shot) - a set of grade-school science questions. - Llama 1 (llama-65b): 57.6 - LLama 2 (llama-2-70b-chat-hf): 64.6 - GPT-3.5: 85.2 - GPT-4: 96.3 HellaSwag (10-shot) - a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models. - Llama 1:…
Earlier quoted context omitted.
Oh that's interesting, the last time I checked GPT-4 I got a much worse result. Yes that answer looks correct to me though it leaves off some later uses in the book series (which is what makes this a slightly tricky question to answer). But in terms of first use it nails it. Did you add the "Mention the events leading up to it, the book number and name, and the chapter title if one exists." part to the prompt? I'm wo…
That is an incredibly accurate answer for a niche question (about the best fantasy series of all time, but still less well known than Lord of the Rings so, niche). How is Bard getting that so accurate? My first thought is that maybe they've had an LLM go over their books training corpus and pre-summarise every single book or series, then trained on those summaries as well. Or maybe they did some semi-automated Q/A ge…
Earlier quoted context omitted.
Still fails my hippo test! > Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the heat of the savannah. In fact, hippos are one of the best swimmers among all land mammals. But that's fine. Most do. Hippos don't swim. They walk or hop/skip at best underwater.
Google says that swimming is "the sport or activity of propelling oneself through water using the limbs". It doesn't constrain the propulsion to only be between the limbs and the water. Seems like pushing against the ground to move through the water fits the definition
Earlier quoted context omitted.
Yeah and I'm saying I don't believe it. I don't know what you're talking about. GPT-4 is the best model out there by significant margin. That's coming from personal usage not benchmarks. A 10% drop in traffic the first month students are out of school is not "losing users quickly" lol. ChatGPT didn't gain public use waving benchmarks around. We didn't even know what they were until GPT-4's release. The vast majority…
Nobody has any evidence here. I'm saying that the incentives are such that the null hypothesis should be the opposite of what you think.
Earlier quoted context omitted.
Evolution is not a mechanism that maximizes it is a set of interrelated elements that operate at entirely different levels (molecular, cellular, individual and species) to pass on those traits from one generation to another that result in (possible) change of future individuals within a population, which may affect the survival of those individuals increasing or decreasing the chances of passing those traits on to th…
Whether the brain is "designed" or not is really irrelevant to the point here. It has nothing to do with whether an objective function is "wrong" or not.
The whole notion of 'objective functions' doesn't enter into it, let alone whether or not they are right or wrong. You can retroactively infer one and say that's what evolution is optimizing for but that's confusing cause and effect.
It's about as sophisticated a view of evolution as a billiard ball simulation of the universe. It just doesn't work that way, it's way too simple a representation to have a chance of modeling the observed complexity. You can try to collapse it into such a simplistic model if you want to explain evolution to a small child. Like a placeholder for something better when they're more equipped to deal with the added complexity. Like when we say electrons move from plus to minus and how the simple semiconductor model allows you to design functional circuitry. But the underlying physics is a lot more complex than that.
Earlier quoted context omitted.
There will be finetunes for other languages just like LLaMAv1
How can you finetune for a new language? Aren't the tokens baked in by the time the model is done training?
The backend does sometimes need a new tokenizer, depending on how its implemented.