at some point this doesn’t make LLMs feel useful. I have to wait 10x as long just so my LLM can have a somewhat higher chance of actually answer my question correctly?
Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
171–180 of 253 posts
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#172Something I do sometimes is: - Have an AI chat model come up with an answer to a problem. - Have it write a report discussing the details of the problem and why it's answer is correct, directed at a person or AI model who has no knowledge of the initial problem or technical field. - Have a second AI model with no knowledge of the problem grade the report, and write it's own report either (a) asking for clarification…
Kagi’s Assistant feature makes this super easy. Just switch assistants and ask them to check the other’s work.
How?
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#173soon there will be AI debates. Different models debating with each other on a topic
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#174Earlier quoted context omitted.
Kagi’s Assistant feature makes this super easy. Just switch assistants and ask them to check the other’s work.
How?
Ask the AI assistant for instructions.
Pretty soon we'll have new acronyms such as "IDKATFAIA" ["I don't know, ask the f'ing AI already"] as we all succumb to the knowledge soup.
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#175I tried something similar when Llama2 came out, pitting two assistants, who each believed the other is the user, against each other. Ultimately, it was the same model talking with itself. The system prompts for both had various instructions to disagree and criticise the opinion of the user. I provided the first message to get things started. Usually, it’s be along the lines of “nuclear proliferation is harmful to hum…
With my own experiments I've also found this. This behavior is very persistent with llms on default hyperparameters and system prompt. Right now I am exploring how to get these models to output more human like interactions and it seems that a very specific and detailed system prompt is very important to get this to work. These systems are VERY sensitive to system prompt and user input. Meaning that the quality of output varies drastically depending on not just the language you use but how its structured, the order of that structure and also other many nuanced things like system prompt plus user input pre conditioning. So far it seems its possible to get to where we need to for this task but lots of exploration needs to be done in finding the way in how to structure the whole system together. This revelation is kind of nuts when you think about it. It basically means, once you find the right words and the order in which they should be structured for the whole system you can get 2x+ improvement in every variable you care about. That's why I am spending some time creating an automated solution to find these things for x model. Its a tedious effort to do manually, but we have the tools to automate its own optimization and calibration efforts.
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#176I kind of want to try something like this at a larger scale in an always-on mode where I have a 'senate' of debate. Rather than responding to prompts on a case by case basis, provide a list of tasks (potentially with deadlines) and let the senate work on them, break off into groups to manage subtasks, challenge results , make suggestions. Even potentially a tree of analysts where suggestions only gets passed up the t…
Yeah, but we'll finally get definitive proof that the government's been hiding super-intelligent axolotls from us all.
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#177Here's some related challenge I'm facing. Maybe someone can help me: I also managed to make AI critique itself and that improved code generation a ton. For a TypeScript backend project that runs with Bun, I tell AI to also generate and run unit tests after every code change suggested by AI. How do you solve the risk of AI writting and executing unit tests with something like `rm -rf /` and wiping your files? Docker w…
No way around it, got to sandbox the whole thing no matter what.
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#178a paper with a similar idea on scaling test time reasoning, this is sorta how all the thinking models work under the hood. https://arxiv.org/abs/2501.19393
Re: Chain of Recursive Thoughts: Make AI think harder by making it argue with itself
#179does it actually make a difference to do M rounds of N vs one round of M*N?
My gut tells me yes. From my own experiments the order and way in which these things are done are important. I think it all is very strongly tied to the attention mechanism.