I was thinking about this just the other day!! Frequently I find myself wanting to interleave thoughts in a conversation, this is the natural way to do so. I just wasn't sure how you'd train it.
Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
11–19 of 19 posts
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#12Hm. I'll have to rethink how I do context management ( https://github.com/Vibecodelicious/context-bonsai-agents#con... )
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#13I am a bit suspicious of these ideas. When I disabled parallel tool calls in my custom gpt5.4 harness, the quality of results went up dramatically. It looks like it's running slower and it probably is for some problems, but it's correct way more often than if I allow parallel calls.
I am perfectly content with a medium-speed golden goose. It seems to be a lot more predictable and happy this way. The business and other developers are already saturated by the serialized technique. Going faster would only serve to distract others at this point.
I think Navy SEALs have an apt slogan here.
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#14Really cool paper and easy to follow. Lots of thoughts (in parallel, hah!) after a first read. I can see many benefits of the parallel streams w/ dynamic systems. Start thinking, fire up a tool call, adjust thinking on the fly. Or add a "clock tick" on one stream, and hope that the model learns how to output something under time constrain. Maybe some "time passing" concept can be had "for free?". Lots and lots of dir…
The potential of tweaking cross-stream attention is a very interesting avenue, like they note in their discussion: "one-way interactions for security, or partial stream isolation for fine-grained privilege control".
Splitting system streams from user streams already decreases likelihood of successful attacks (e.g., prompt injection) in their research, and that is - as they say - using the dense attention patterns between streams.
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#15I think 'actual parallelism' is a vastly easier and more fruitful way to get better performance out of these kinds of systems, compared to pushing against single-threaded faster generation. Tool calling and responses are often embarrassingly parallel. Code generation tasks naturally have a dependency tree that can be unrolled into a fixed budget of parallelism. Tasks can be hierarchically decomposed into subtasks. It…
I do this kind of parallelism with a little merge request tool I slopped together. I spin up multiple small agents and assign them specific code review tasks (security, coding standards, etc.) and have it spit out a gitlab API draft json object with code examples for the MR I can deterministically validate against. If it fails to insert code examples (depending on the task) and the proper json object schema, I have "ask it to try again" logic in place.
Works fine, forcing LLMs to output parsable responses is a good workaround to get them to do what you want until they improve. It also allows you to use the fast models (ex. I spin up the Gemini 3.1 flash lite model for these tasks) to have these tasks done in seconds rather than minutes.
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#16[dead]
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#17I think 'actual parallelism' is a vastly easier and more fruitful way to get better performance out of these kinds of systems, compared to pushing against single-threaded faster generation. Tool calling and responses are often embarrassingly parallel. Code generation tasks naturally have a dependency tree that can be unrolled into a fixed budget of parallelism. Tasks can be hierarchically decomposed into subtasks. It…
I do this kind of parallelism with a little merge request tool I slopped together. I spin up multiple small agents and assign them specific code review tasks (security, coding standards, etc.) and have it spit out a gitlab API draft json object with code examples for the MR I can deterministically validate against. If it fails to insert code examples (depending on the task) and the proper json object schema, I have "…
Similar to your method
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#18just use the same context cache and a separate completion assuming elastic resources
Re: Multi-Stream LLMs: new paper on parallelizing/separating prompts, thinking, I/O
#19I am a bit suspicious of these ideas. When I disabled parallel tool calls in my custom gpt5.4 harness, the quality of results went up dramatically. It looks like it's running slower and it probably is for some problems, but it's correct way more often than if I allow parallel calls. I am perfectly content with a medium-speed golden goose. It seems to be a lot more predictable and happy this way. The business and othe…
i don't think that has anything to do with this paper. isn't parallel tool calling just "assemble several tool calls at once"