Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

app.uniclaw.ai

Post reply on HN
Print view View original

81–90 of 97 posts

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#81

Post by throwa356262 »

skysniper wrote: post #40
mgw wrote: post #36

Missing from the comparison is MiMo V2 Flash (not Pro), which I think could put up a good fight against Step 3.5 Flash. Pricing is essentially the same: MiMo V2 Flash: $0.09/M input, $0.29/M output Step 3.5 Flash: $0.10/M input, $0.30/M output MiMo has 41 vs 38 for Step on the Artificial Analysis Intelligence Index, but it's 49 vs 52 for Step on their Agentic Index.

I will try and add it. But I doubt it works well because Mimo V2 Pro is beaten by stepfun even at performance leaderboard (price is not a factor in this leaderboard), so I expect MiMo V2 Flash to perform even worse.

Interesting, I found the pro version to be very capable.

If stepfun is even better, then Chinese models are getting really good.

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#82

Post by throwa356262 »

skysniper wrote: post #18

another thing from the bench I didn't expect: gemini 3.1 pro is very unreliable at using skills. sometimes it just reads the skill and decide to do nothing, while opus/sonnet 4.6 and gpt 5.4 never have this issue.

Gemini 2.5 pro was the best Gemini, it has gone downhill since

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#83

Post by sunaookami »

admiralrohan wrote: post #51
sunaookami wrote: post #33

Tried the free version on OpenRouter with pi.dev and it's competent at tool calling and creative writing is "good enough" for me (more "natural Claude-level" and not robotic GPT-slop level) but it makes some grave mistakes (had some Hanzi in the output once and typos in words) so it may be good with "simple" agentic workflows but it's definitely not made for programming nor made for long writing.

What kind of creative writing are you doing? Fiction or non-fiction like blog posts?

Fiction. One of my "benchmarks" is giving the model a bunch of (self-made) text and having it simulate a 4chan thread about it. This tests tool use (calling the APIs), some skills, censorship and general creativity. Some models refuse every new turn after reading real 4chan threads ;) Claude is especially good at this surprisingly while GPT fails spectacularly and Gemini is just lazy (and barely usable since it's constantly overloaded). Qwen (coder-model from Qwen CLI, so Qween 3.5) is also very good but sadly not usable in Pi (they detect and block calls outside their CLI).

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#85

Post by johndough »

skysniper wrote: post #63
johndough wrote: post #41

Earlier quoted context omitted.

I would have liked aggregated results instead. Expanding 300 tables is a bit tiresome. But I guess that is easy with AI now. Here is a scatter plot of quality vs duration https://i.imgur.com/wFVSpS5.png and quality vs cost https://i.imgur.com/fqM4edw.png But I just noticed that my plot is meaningless because it conflates model quality with provider uptime. Claude Haiku has a higher average quality than Claude Opus, w…

i added native plot and stats for aggregated results, on arena page. please check it out!

Nice! It would be even better if the model name was shown by default instead of having to hover, but I got the information that I wanted. In case you should be concerned about the aesthetics with too many model names, I can recommend the adjustText library in Python, which makes it so that labels do not overlap. Something similar probably exists in JS (or an LLM can just translate the relevant bits).

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#86

Post by wilj »

wg0 wrote: post #78
kdasme wrote: post #75

Earlier quoted context omitted.

Minimax M2.7 is similar to sonnet in my tests. This is the first non OAI/Anthropic model I use for coding. It does require more steering, though.

More steering than Sonnet? What is your experience?

I'm about 2 days into transitioning, using MiMo V2 Pro in place of Opus and MiniMax M2.7 in place of Sonnet.

I'm finding that the extra "hand holding" that MiMo and MiniMax need isn't really "extra." The Anthropic models happily agree to a plan and then do something else entirely way too often.

With MiMo and MiniMax I'm just spreading the attention throughout the day instead of big spikes of frustration figuring out where Claude went off the rails.

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#87

Post by tarruda »

lostmsu wrote: post #48
tarruda wrote: post #14

Earlier quoted context omitted.

Since that discussion, they released the base model and a midtrain checkpoint: - https://huggingface.co/stepfun-ai/Step-3.5-Flash-Base - https://huggingface.co/stepfun-ai/Step-3.5-Flash-Base-Midtra... I'm not aware of other AI labs that released base checkpoint for models in this size class. Qwen released some base models for 3.5, but the biggest one is the 35B checkpoint. They also released the entire training pipel…

Tuned Qwen 3.5 27B beats Step 3.5 on almost all benchmarks, so the point about the size class is moot.

Benchmarks don't tell the whole story. For one-shot coding tasks, I found Step 3.5 Flash to be stronger even than Qwen 3.5 397B.

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#88

Post by hypercube33 »

skysniper wrote: post #18

another thing from the bench I didn't expect: gemini 3.1 pro is very unreliable at using skills. sometimes it just reads the skill and decide to do nothing, while opus/sonnet 4.6 and gpt 5.4 never have this issue.

I used sonnet and opus 4.6 for a month and it flat out ignored skills and rules and when asked it said it knew better or was lazy.

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#89

Post by Aerroon »

p1necone wrote: post #70
Aerroon wrote: post #68

Earlier quoted context omitted.

If they use OpenRouter pricing then the Qwen3.5 models are going to be poor value. The Qwen3.5 27B model on OR is $1.56/million tokens out (it used to be $2.4/mil). Meanwhile Minimax M2.7 (a much larger model) is $1.2/mil out. The smaller and medium tier Qwen3.5 models are only really cost effective if you run them yourself.

Is Minimax M2.7 better than Qwen3.5 27B, or is it just bigger?

Yes, it's significantly better.

Re: StepFun 3.5 Flash is #1 cost-effective model for OpenClaw tasks (300 battles)

#90

Post by wg0 »

wilj wrote: post #86
wg0 wrote: post #78

Earlier quoted context omitted.

More steering than Sonnet? What is your experience?

I'm about 2 days into transitioning, using MiMo V2 Pro in place of Opus and MiniMax M2.7 in place of Sonnet. I'm finding that the extra "hand holding" that MiMo and MiniMax need isn't really "extra." The Anthropic models happily agree to a plan and then do something else entirely way too often. With MiMo and MiniMax I'm just spreading the attention throughout the day instead of big spikes of frustration figuring out…

Thank for responding. So you are using MiMo V2 Pro to plan and then asking MiniMax M2.7 to read that plan file and execute? Or how the workflow looks like?

Pi/Opencode/Kilocode? Just curious.

I am using Opencode mostly and thinking to abandon Copilot so looking for something similar.

Post reply on HN