Viewing profile — george_123

1) when the blog was released, TGI didn’t support paged attention, 2) many people don’t even know about TGI to reduce inference costs.

A lot of AWS services (especially SageMaker) aren’t very good in customer experience. People buy them for nominal capabilities and AWS core bread and butter — short-term and long-t…

this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..

This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores f…

This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores f…