Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
1–10 of 61 posts
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#2It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#3It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
Nothing is free in this world.
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#4It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
I've only read the abstract but they don't mention quantizing the weights or otherwise trying to shrink the model in any way.
They're claiming to be able to efficiently run larger models without loading the entire thing into GPU memory. If they're using the same weights, the same architecture and just using tensor parallel operations to perform the forward pass that would imply no loss in quality.
I'm sure there are trade-offs but they're not clear by just looking at the abstract.
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#5It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
From what I get skimming through the article the main cost is speed of token generation (token latency). You can always run a large model by reading directly from the disk and not care much about ram; but it is very slow. They try to improve that aspect doing some optimisations, but it is still definitely slower than using ram or vram.
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#6It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
[deleted]
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#7It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
It's not cost-free. It comes at the cost of greatly increased latency. 29.9 seconds per token with Llama 3.1-70B. This is from Table 1 (pg 8) of the paper.
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#8It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
It's not cost-free. It comes at the cost of greatly increased latency. 29.9 seconds per token with Llama 3.1-70B. This is from Table 1 (pg 8) of the paper.
Ah the disk swap method
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#9Earlier quoted context omitted.
It's not cost-free. It comes at the cost of greatly increased latency. 29.9 seconds per token with Llama 3.1-70B. This is from Table 1 (pg 8) of the paper.
Ah the disk swap method
Is there any predictability/patterns for neuron/layer activation? If so, would it be reasonable to have a second tiny model that specifically tries to predict activation and preemptively swap those into memory?
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#10It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.
From what I get skimming through the article the main cost is speed of token generation (token latency). You can always run a large model by reading directly from the disk and not care much about ram; but it is very slow. They try to improve that aspect doing some optimisations, but it is still definitely slower than using ram or vram.
Table 3 directly refutes this* and claims 0 tradeoffs.**
Below that, they indicate that a key part of the implementation is loading weights from disk before they're needed using a separate thread.***
* maybe I'm missing something though, someone please triple check :)
** ttft (time to first token) and s/token (seconds per token) are both lower than any alternative in all cases.
*** "daemon thread asynchronously preloads the weights"