Viewing profile — weichiang

say using A10G ~$1.2/hr and with full utilization on vllm 112 reqs/min => per req ~$0.00018 versus gpt-3.5 turbo $0.002 per 1k token

the cost would be depending on GPU type/serving system/traffic pattern. check out some throughput comparison from vllm's blog post https://vllm.ai/ if you serve 7B on cost-optimize…

leaderboard link: https://chat.lmsys.org/?leaderboard

Surprised to learn StableLM is worse than plain LLaMA. link to their leaderboard: leaderboard.lmsys.org

I asked GPT4All one of Vicuna's benchmark questions: "What if the Internet had been invented during the Renaissance period?" Check out their responses: https://imgur.com/a/mPrdZ1W …

See the original Vicuna post: https://news.ycombinator.com/item?id=35378683

just tried it worked okay for me tho. > Can you name all of the soccer players whose names contain a Z > Here are all the soccer players whose names contain a "Z" that I could find…