A guide to open-source LLM inference and performance
1–10 of 16 posts
Re: A guide to open-source LLM inference and performance
#2Likely trending on home page since this is directly relevant to LLM costs, i.e., questions like "how much would it cost to rebuild ChatGPT from scratch".
Re: A guide to open-source LLM inference and performance
#3Likely trending on home page since this is directly relevant to LLM costs, i.e., questions like "how much would it cost to rebuild ChatGPT from scratch".
highly relevant question today
Re: A guide to open-source LLM inference and performance
#4love the deep dive here
Re: A guide to open-source LLM inference and performance
#5Likely trending on home page since this is directly relevant to LLM costs, i.e., questions like "how much would it cost to rebuild ChatGPT from scratch".
[deleted]
Re: A guide to open-source LLM inference and performance
#6That’s really detailed explanation. Can we do something like this for M1 ultra/M2 ultra/M3 max with large RAM ?
Re: A guide to open-source LLM inference and performance
#7Re: A guide to open-source LLM inference and performance
#8That’s really detailed explanation. Can we do something like this for M1 ultra/M2 ultra/M3 max with large RAM ?
Absolutely. Looks like the M1 Ultra has 800GB/s of memory bandwidth and ~20 TFLOPS of compute.
The same calculations from the post should hold, except with these new values.
Re: A guide to open-source LLM inference and performance
#9Thanks a lot for the material Varun, neat presentation with exhaustive computations that make it easy to follow. Question on the serving part: vLLM, Deepspeed, TensorRT-LLM... ? Thanks!
Re: A guide to open-source LLM inference and performance
#10Thanks a lot for the material Varun, neat presentation with exhaustive computations that make it easy to follow. Question on the serving part: vLLM, Deepspeed, TensorRT-LLM... ? Thanks!
Thanks!
vLLM for quick set up, TRT-LLM for best performance. Both available on https://baseten.co/.