Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide
1–7 of 7 posts
Re: Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide
#2[deleted]
Re: Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide
#3Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128G RAM for ~50 euro monthly.
Re: Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide
#4Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128G RAM for ~50 euro monthly.
Is inference speed an issue for you?
Re: Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide
#5Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128G RAM for ~50 euro monthly.
usually performance takes a hit with quantization. are you getting quality responses?
Re: Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide
#6Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128G RAM for ~50 euro monthly.
usually performance takes a hit with quantization. are you getting quality responses?
Since llama3, yes, quite satisfying.