Viewing profile — slimesli

for llama 3-8b (smaller model) you need 8GB/16GB of memory: minimum/recommended || for llama 3-70b (larger model) you need 64GB/96GB of memory what kind of memory? there are 3 type…

It's not a fundamental limit. Google palm had 540B parameters as dense model. But it's a practical limit because models with over 1T would be extremely slow even on newest gpus. Ev…

Because of memory bandwidth. H100 has 3350gB/s of bandwidth, more gpus will give you more memory but not bandwidth. If you load 175b parameters in 8bit then you can get theoretical…