11–20 of 34 posts
The problem with these Open weights LLMs hosted by these provider is that we don't what's the precision of the LLM, that makes a huge difference in the speed and cost (compute). I think Together recently introduced a different price tier based on precision but otherwise it is usually dark.
Something Open Router get's close to that https://openrouter.ai/models/meta-llama/llama-3.1-405b-instr...
Nice I built something similar https://huggingface.co/spaces/Whiteshadow12/llm-pricing-calc... I like your charting, many have taken this task and then lose interest. similar other tools for inspiration https://llmprices.dev/ https://www.llmpricing.app/ What no one is doing is focusing on GPUs, what is the cost of running L3-8B on an A100 or H100 per second.
What sets them apart is that they have speed and latency as well.
Nice I built something similar https://huggingface.co/spaces/Whiteshadow12/llm-pricing-calc... I like your charting, many have taken this task and then lose interest. similar other tools for inspiration https://llmprices.dev/ https://www.llmpricing.app/ What no one is doing is focusing on GPUs, what is the cost of running L3-8B on an A100 or H100 per second.
Yeah we want to do exactly this, benchmark and add more data from differnt gpus/cloud providers, will appreciate your help a lot! There are many inference engines which can be tested and updated to find best inference methods
It's a lot of work, your target users is companies that use Runpod and AWS/GCP/Azure, not Fireworks and Together, they are in the game of selling tokens, you are selling the cost of running seconds on GPUs.
Litellm maintains list of model infos here: https://github.com/BerriAI/litellm/blob/main/model_prices_an... Simple UI to search: https://models.litellm.ai/?q=llama3
Earlier quoted context omitted.
Yeah we want to do exactly this, benchmark and add more data from differnt gpus/cloud providers, will appreciate your help a lot! There are many inference engines which can be tested and updated to find best inference methods
Goodluck, companies would love that. Don't get depressed unlike my tool I think you should charge, that might keep you motivated to keep doing the work. It's a lot of work, your target users is companies that use Runpod and AWS/GCP/Azure, not Fireworks and Together, they are in the game of selling tokens, you are selling the cost of running seconds on GPUs.
So overall not cheaper running these yourself compared to using GPT-4o mini.
Without any quantization our current price is 30cts ingest and 50cts output per million tokens. [1]
We built the cheapest Llama 3.1 70B inference API, specialized for tasks that are not time sensitive (ie. batch processing jobs for example). Without any quantization our current price is 30cts ingest and 50cts output per million tokens. [1] 1: https://withexxa.com/#pricing
Litellm maintains list of model infos here: https://github.com/BerriAI/litellm/blob/main/model_prices_an... Simple UI to search: https://models.litellm.ai/?q=llama3