61–70 of 78 posts
Earlier quoted context omitted.
Is it because Pi’s default system prompt is so simple?
of course it is; these entire systems are about priming the LLM to perform X task; if it's a coding harness, giving it very little priming for a task is going to make it grasp at straws.
We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting vi…
Earlier quoted context omitted.
The catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.
Pi has a subscription flow for Claude Code. Are they goading people to get banned or did they figure out a way to work with the subscription?
> as we aggressively adopt AI for engineering Why do we need to aggressively adopt things rather than thoughtfully adopt things? It sounds like they are probably punching AI and engineers in the process
What if other competing companies adopt it aggressively and push out features faster and get more market share? What if pushing features faster brings more money because users like the features?
Have we seen a company fail because they're not adopting AI as much as their competitors?
Its shocking how cost per token does not correlate with cost per task, it's wild to see opus and glm nearby on $ per task axis
Wow! It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results. Definitely a few surprises here: 1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25…
The catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.
It seems that pass rate decreases with effort increase, on GPT5.5? This is highly counter-intuitive and I don't see any explanation, any idea why they'd get this result?
How is Pi so efficient? You'd think agent harness made by model makers would perform better.