71–80 of 96 posts
when will the llm race peak? have we peaked already?
If you’re curious, you can check the progress of many open source LLM’s and how they perform on various evals here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
Earlier quoted context omitted.
You must have missed the memo... It's now super easy to extend the context of 2k llama models to 8k, 16k, or even 32k with just a small fine tune and a tweak to the code. You still need the memory to be able to go that high, but it's totally doable.
What memo/paper does this?
Earlier quoted context omitted.
That blog post demonstrates that it's not "easily" finetuneable, just possible to finetune. There's many technical considerations even beyond hardware (dataset formatting, training hyperparameter nuances) that do not make it accessible to the newbie experimenting with LLMs. It's a rabbithole, and unfortunately there's no good shortcuts.
Why have there been thousands of overnight AI/GPT startups and products in the last few months and NOT a single simple intuitive "fine tuning wizard" app? That seems like such an obvious glaring gap.
Vapourware GPT startup inc is valued at $2bn the afternoon after you form the company and buy your first macbook.
Actual usage of Ai, fine tuning etc. I can offer you $100,000 for 30% of your company if you can demonstrate a fully working product.
> The training recipe and model architecture follow LLaMA This is huge. MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it's going to catch on much faster than any other small model.
XGen-7B is probably the superior 7B model, it's trained on more tokens and a longer default sequence length (although both presumably can adopt SuperHOT (Position Interpolation) to extend context), but larger models still probably perform better on an absolute basis.
If someone could elucidate on what these phrases signify, I'd be very grateful: 1) 7B foundational model 2) 8K length 3) 1.5T tokens
- 7B means 7 billions parameters.
- 8K length means the size of input/output is 8K tokens.
- 1.5T tokens mean the training set has 1.5T tokens.
A: What's a parameter?
Q: More parameters your model has, more complex relationship it can represent. For example let's say you have a function f(x). This is a 2-parameter model:
f(x) = ax + b
This is a 4 parameter model:
f(x) = ax^3 + bx^2 + cx + d
As you can see as the number of parameters grows, the function is able to represent more complex relationship between f(x) and x.
A: What's a token?
Token is a way to encode text, like ASCII or Unicode. Unlike Unicode, tokenizor usually favors common combinations of alphabets. For example, "the" is a single token for GPT-3 tokenizor, but "eht" is two tokens (e and ht).
* Note that the number of parameters is more like an "upper limit" of the model's capabilities. If your a, b, c, d are just random shit, it's still a 4-parameter model, but it's still useless. The whole concept of "training" is just "finding the best parameters".
Earlier quoted context omitted.
But if you don't know a certain amount about a subject already you won't know when it's lying to you. That would probably be the case here.
It's basically what an interactive dialogue with Wikipedia would look like, which is still a darned useful thing.
Earlier quoted context omitted.
Nobody doing xB models is participating in any AI races, at this point those are useless toys with garbage output.
How far we've come from GPT2...
If someone could elucidate on what these phrases signify, I'd be very grateful: 1) 7B foundational model 2) 8K length 3) 1.5T tokens
"7B" refers to the number of parameters or weights for a model. For a specific model, the versions with more parameters take more compute power to train and perform better.
A foundational model is the part of a ML model that is "pretrained" on a massive data set (and usually is the bulk of the compute cost). This is usually considered the "raw" model after which it is fine-tuned for specific tasks (turned into a chatbot).
"8K length" refers to the Context Window length (in tokens). This is basically an LLM's short term memory - you can think of it as its attention span and what it can generate reasonable output for.
"1.5T tokens" refers to the size of the corpus of the training set.
In general Wikipedia (or I suppose ChatGPT 4/Bing Chat with Web Browsing) is a decent enough place to start reading/asking basic questions. I'd recommend starting here: https://en.wikipedia.org/wiki/Large_language_model and finding the related concepts.
For those going deeper, there are lot of general resources lists like https://github.com/Hannibal046/Awesome-LLM or https://github.com/Mooler0410/LLMsPracticalGuide or one I like, https://sebastianraschka.com/blog/2023/llm-reading-list.html (there are a bajillion of these and you'll find more once you get a grasp on the terms you want to surf for). Almost everything is published on arXiv, and most is fairly readable even as a layman.
For non-ML programmers looking to get up to speed, I feel like Karpathy's Zero to Hero/nanoGPT or Jay Mody's picoGPT https://jaykmody.com/blog/gpt-from-scratch/ are alternative/maybe a better way to understand the basic concepts on a practical level.
when will the llm race peak? have we peaked already?
Earlier quoted context omitted.
7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion. The metrics are good though, perhaps placing this closer to 13B. And 8K context is huge . When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM…
would appreciate resources on fine tuning