Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source LLM provider price comparison

github.com

Post reply on HN
Print view View original

31–34 of 34 posts

Re: Show HN: Open-source LLM provider price comparison

#31

Post by Deathmax »

alextttty wrote: post #8
yujonglee wrote: post #3

Litellm maintains list of model infos here: https://github.com/BerriAI/litellm/blob/main/model_prices_an... Simple UI to search: https://models.litellm.ai/?q=llama3

Thank you! yeah great source. Do they track throughput for open source models? and inference engines Thats kind of data I want to find as well

For throughput data, well, you need to actually run prompts to gather the data which racks up costs fast and performance can vary based on input prompt lengths. The two sources I use are OpenRouter's provider breakdown [1] and Unify's runtime benchmarks [2].

[1]: https://openrouter.ai/models/meta-llama/llama-3.1-70b-instru...

[2]: https://unify.ai/benchmarks/llama-3.1-70b-chat

Re: Show HN: Open-source LLM provider price comparison

#33

Post by agcat »

Whiteshadow12 wrote: post #13
alextttty wrote: post #7

Earlier quoted context omitted.

Yeah we want to do exactly this, benchmark and add more data from differnt gpus/cloud providers, will appreciate your help a lot! There are many inference engines which can be tested and updated to find best inference methods

Goodluck, companies would love that. Don't get depressed unlike my tool I think you should charge, that might keep you motivated to keep doing the work. It's a lot of work, your target users is companies that use Runpod and AWS/GCP/Azure, not Fireworks and Together, they are in the game of selling tokens, you are selling the cost of running seconds on GPUs.

This is true especially if you are deploying custom or fine-tuned models. Infact, for my company i also ran benchmark tests where we tested cold-starts, performance consistency, scalability, and cost-effectiveness for models like Llama2 7Bn & Stable Diffusion across different providers - https://www.inferless.com/learn/the-state-of-serverless-gpus... Can save months of evaluation time. Do give it a read.

P.S: I am from Inferless.

Re: Show HN: Open-source LLM provider price comparison

#34

Post by edgoode »

alextttty wrote: post #10
amrrs wrote: post #5

The problem with these Open weights LLMs hosted by these provider is that we don't what's the precision of the LLM, that makes a huge difference in the speed and cost (compute). I think Together recently introduced a different price tier based on precision but otherwise it is usually dark.

Exactly, always best to rely on your hardware, we need to collect/add more data from self hosted models on different gpus/clouds to compare

We have 15+ clouds you can try on our platform if you're looking for a place to compare inference engines

Email me at ed at shadeform dot ai if we can help

Post reply on HN