11–20 of 134 posts
Results seem mostly noise to me. One eval per model, in a large problem space (i.e. a problem which requires many attempts to solve well).
What is /goal?
I love that we have this on one hand and me cleaning up catastrophic CSS made by Sol on the other. Then again, maybe CSS is the ultimate benchmark.
What is /goal?
I love that we have this on one hand and me cleaning up catastrophic CSS made by Sol on the other. Then again, maybe CSS is the ultimate benchmark.
Offering freelance estimates for CSS design changes before frameworks were around was a problem.
/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now: 1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed) 2. What time is it? 3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forwa…
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction. For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact