21–30 of 32 posts
So I can run a 4bit quantization of original Llama 8B on my laptop. It pretty much uses up all of my 6GB Nvidia card. Will I be able to just run this model on the same laptop with an increased context window?
So I can run a 4bit quantization of original Llama 8B on my laptop. It pretty much uses up all of my 6GB Nvidia card. Will I be able to just run this model on the same laptop with an increased context window?
Nope. You can run the model fine but if you actually want to take advantage of the big context window the memory usage will grow enormously. For the 256k they already require 64GB... So for this I guess 256GB? Source: https://ollama.com/library/dolphin-llama3:256k > Note: using a 256k context window requires at least 64GB of memory. If I run that 256k model with simple typed prompts it behaves the same as the normal…
Earlier quoted context omitted.
Hi, Leo, chief scientist @ Gradient, here. We've been eagerly awaiting the release of RULER's code ourselves! As mentioned below, we wanted to release a model to the community asap, and have plans already for further fine-tuning & more sophisticated evals. If you have other suggestions, I'd be happy to chat further.
Hi! Unless I'm missing something, they did add the eval scripts to that repo 4 days ago.
Earlier quoted context omitted.
Nope. You can run the model fine but if you actually want to take advantage of the big context window the memory usage will grow enormously. For the 256k they already require 64GB... So for this I guess 256GB? Source: https://ollama.com/library/dolphin-llama3:256k > Note: using a 256k context window requires at least 64GB of memory. If I run that 256k model with simple typed prompts it behaves the same as the normal…
can you select a context length that fits in your GPU though? I suppose even a 128k model would be more than enough for almost everyone running these models on their own hardware.
Wow there was already a 256k version (dolphin). 1M is insane. Be aware you need a lot of memory though
With 144gb of GPU memory, The most I can load for llama3 is 232k.
Just wondering. I'll never have that kind of resources (well not in the next 5 years) but just trying to put it into perspective..
Will you also do a 70B finetune? Thanks.
Earlier quoted context omitted.
can you select a context length that fits in your GPU though? I suppose even a 128k model would be more than enough for almost everyone running these models on their own hardware.
No you can't right now. Hopefully they will add this to ollama.
Earlier quoted context omitted.
With 144gb of GPU memory, The most I can load for llama3 is 232k.
Which llama3 is that? 8b or 70b? And what kind of quantisation? Just wondering. I'll never have that kind of resources (well not in the next 5 years) but just trying to put it into perspective..