Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
1–10 of 11 posts
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#2Really cool!
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#3[flagged]
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#4This is sick!
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#5Oh boy love it!
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#6I'll be honest that I've never actually considered tokens per second as something to focus on for my projects, I'm much more concerned with quality of the output then quantity.
Is 500 tok/s on Gemma 7B a gamechanger? or is this more just an advertisement for mystic.ai?
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#7I'll be honest that I've never actually considered tokens per second as something to focus on for my projects, I'm much more concerned with quality of the output then quantity. Is 500 tok/s on Gemma 7B a gamechanger? or is this more just an advertisement for mystic.ai?
There’s a percentage of users that do care about token generation speed as they chain multiple API calls. The performance is all thanks to TensorRT-LLM, Mystic takes care of the engineering of getting a scalable endpoint out of it, i.e, not having to manage your infra.
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#8I'll be honest that I've never actually considered tokens per second as something to focus on for my projects, I'm much more concerned with quality of the output then quantity. Is 500 tok/s on Gemma 7B a gamechanger? or is this more just an advertisement for mystic.ai?
[dead]
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#9This is great - taking scalability to the next level
Re: Deploy Gemma 7B with TensorRT-LLM and achieve > 500 tok/s
#10I'll be honest that I've never actually considered tokens per second as something to focus on for my projects, I'm much more concerned with quality of the output then quantity. Is 500 tok/s on Gemma 7B a gamechanger? or is this more just an advertisement for mystic.ai?
the latency to serve the end-user esp with chained calls impacts ux quality