11–20 of 48 posts
Another thing is if you're using the subscriptions with OpenAI or Anthropic you get an order of magnitude discount relative to the per-token price. So you need to move their models ~10x to the left on the plots to get a fair comparison.
Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.
> Why AA’s plot is misleading > The first issue I have with it is that it uses a logarithmic scale on the cost axis. Using a log scale is the only way to make you spot the difference between a model that costs $0.015 per task and one that costs $0.032, while the same plot contains a model that costs $3.69 — almost 250 times as expensive. However, the net result is that the viewers can no longer appreciate the immensi…
Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.
there's a fun Excel artifact where it auto-selects the 'relevant' range with no adjustment for how proportionally close to 0 the values are - a professional researcher publishing to a journal should know better (and should be ridiculed for not incorporating best practices) but for a personal blog by an SWE this really isn't the worst sin
[0] https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have...
Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.
to be fair, you don't need to start the y-axis at zero [0] but for some of the graphs where the lowest value is close to 0 the best practice is to do so there's a fun Excel artifact where it auto-selects the 'relevant' range with no adjustment for how proportionally close to 0 the values are - a professional researcher publishing to a journal should know better (and should be ridiculed for not incorporating best prac…
Just look at the first chart: the distance between Fable 5.1 and Sol is <5%, but it looks like 25 or 30%.
Nebari is officially listed as a JATIC product as part of the next-gen toolchain supporting DoD AI development.
Are we officially ~one degree of Kevin Bacon from the DoD endorsing running Chinese OSS models because they're self-hosted and we're all too dumb to tell the difference?
https://openteams.com/open-source-isnt-the-real-risk-in-nati...
This looks great! I also think speed should be part of the metric (i.e. how long does the model take to actually solve a task). For me, I prefer to run expensive models such as Sol on light reasoning, which usually gives me good answers with quick responses. For my style of coding (quick back-and-forths and corrections) it makes a big difference if a model comes back in 1-2 minutes compared to 5-10, and I am happy to…
As someone who only needs AI for a couple of tasks per day, I don't really care how much it costs, especially when subscriptions are subsidized. I want to filter by speed (eg, max task time feels great and are affordable for infrequent tasks.
Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.