Did something kind of similar (needs a little updating, but I find it still useful) for price vs performance on the LMSYS leaderboard.[0][1]
Show HN: Open-source LLM provider price comparison
21–30 of 34 posts
Re: Show HN: Open-source LLM provider price comparison
#22LLMs are really commodities now: https://www.lycee.ai/blog/why-large-language-models-are-a-co...
Re: Show HN: Open-source LLM provider price comparison
#23Nice I built something similar https://huggingface.co/spaces/Whiteshadow12/llm-pricing-calc... I like your charting, many have taken this task and then lose interest. similar other tools for inspiration https://llmprices.dev/ https://www.llmpricing.app/ What no one is doing is focusing on GPUs, what is the cost of running L3-8B on an A100 or H100 per second.
Thanks for bringing llmprices.dev to my attention. I have also a comparison page for models hosted on OpenRouter (https://minthemiddle.github.io/openrouter-model-comparison/), I do comparison via regex (so "claude-3-haiku(?!:beta)|flash" will show you haiku, but not haiku-beta vs flash.
I wish that OpenRouter would also expose the amount of output tokens via API as this is also an important criteria.
Re: Show HN: Open-source LLM provider price comparison
#24LLMs are really commodities now: https://www.lycee.ai/blog/why-large-language-models-are-a-co...
Indeed!
Re: Show HN: Open-source LLM provider price comparison
#25So overall not cheaper running these yourself compared to using GPT-4o mini.
I don't think it ever has been. The reason you run them yourself is for the privacy and customizability
Re: Show HN: Open-source LLM provider price comparison
#26Meta comment: he could make an Action that generates the images.
Re: Show HN: Open-source LLM provider price comparison
#27Meta comment: he could make an Action that generates the images.
Made action to gen csvs + json
Working on the mui data table frontend right now
Re: Show HN: Open-source LLM provider price comparison
#28The problem with these Open weights LLMs hosted by these provider is that we don't what's the precision of the LLM, that makes a huge difference in the speed and cost (compute). I think Together recently introduced a different price tier based on precision but otherwise it is usually dark.
I have realized that in my benchmarks [1]. Llama 3 was significantly better than Llama 3.1, which was puzzling.
Then I realized, that I changed the provider. And the new one quantized Llama 3.1 with fp8.
Then I tried Hyperbolic [2], because they offer the model in different quantizations. As result, Llama 3.1 was better than Llama 3 or at least on par.
Re: Show HN: Open-source LLM provider price comparison
#29LLMs are really commodities now: https://www.lycee.ai/blog/why-large-language-models-are-a-co...
Indeed!
and it changes the dynamics of the generative AI space completely ! absolutely exciting to watch. I am bullish on generative AI even if I think scaling laws will generate diminishing returns going forward.
Re: Show HN: Open-source LLM provider price comparison
#30Everyone should know that every single model runs differently on every system and so to decide which is best literally requires you to go through the painstaking process of running inference with each provider and then deciding. The price of inference is not sufficient to decide where to run your models.