161–170 of 255 posts
Earlier quoted context omitted.
Are there some regulatory reasons why it would not be available? It seems weird if Google would intentionally block users merely to block them.
I think there are still some pretty onerous laws about French localization of products and services made available in the French-speaking part of Canada. Could be that...
Earlier quoted context omitted.
Not really, looks like a ~40B class model which is very runnable.
It's actually ~13B class at runtime. 2B for attention is shared across each expert and then it runs 2 experts at a time. So 2B for attention + 5Bx2 for inference = 12B in RAM at runtime.
Earlier quoted context omitted.
What does expert mean in this context?
It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.
Earlier quoted context omitted.
Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?
More or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possi…
Earlier quoted context omitted.
MistralLite is already 32K, and Yi 200K actually works pretty well out to at least 75K (the most I tested)
What kind of tests did you run out to that length? (Needle in haystack, summarization, structured data extraction, etc) What is the max number of tokens in the output?
MistralLite is basically overfit to summarize and retrieve in its 32K context, but its extremely good at that for a 7B. Its kinda useless for anything else.
Yi 200K is... smart with the long context. An example I often cite is a Captain character in a story I 'wrote' with the llm. A Yi 200K finetune generated a debriefing for like 40K of context in a story, correctly assessing what plot points should be kept secret and making some very interesting deductions. You could never possibly do that with RAG on a 4K model, or even models that "cheat" with their huge attention like Anthropic.
I test at 75K just because that's the most my 3090 will hold.
Earlier quoted context omitted.
UNA: Uniform Neural Alignment. Haven't u noticed yet? Each model that I uniform, behaves like a pre-trained.. and you likely can fine-tune it again without damaging it. If you chatted with them, you know .. that strange sensation, you know what is it.. Intelligence. Xaberius-34B is the highest performer of the board, and is NOT contaminated.
How much data do you need for UNA? Is a typical fine tuning dataset needed or can you get away with less than that?
Earlier quoted context omitted.
UNA: Uniform Neural Alignment. Haven't u noticed yet? Each model that I uniform, behaves like a pre-trained.. and you likely can fine-tune it again without damaging it. If you chatted with them, you know .. that strange sensation, you know what is it.. Intelligence. Xaberius-34B is the highest performer of the board, and is NOT contaminated.
How much data do you need for UNA? Is a typical fine tuning dataset needed or can you get away with less than that?
Earlier quoted context omitted.
Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?
More or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possi…
We clearly see that Mistral-7B is in some important, representative respects (eg coding) superior to Falcon-180B, and superior across the board to stuff like OPT-175B or Bloom-175B.
"Well trained" is relative. Models are, overwhelmingly, functions of their data, not just scale and architecture. Better data allows for yet-unknown performance jumps, and data curation techniques are a closely-guarded secret. I have no doubt that a 7B beating our best 60-70Bs is possible already, eg using something like Phi methods for data and more powerful architectures like some variation of universal transformer.
Earlier quoted context omitted.
I will take weights over docs. Its does remind me how some Google employee was bragging that they disclosed the weights for the Gemini, and only the small mobile Gemini, as if that's a generous step over other companies.
I don't think that's true, because quite simply, they have not. I am 100% in agreement with your viewpoint, but feel squeamish seeing an un-needed lie coupled to it to justify it. Just so much Othering these days.
https://twitter.com/zacharynado/status/1732425598465900708
(Alt: https://nitter.net/zacharynado/status/1732425598465900708)
That is fair though, this was an impulsive addition on my part.