41–50 of 70 posts
Funny thing is the structured output in the last example. ``` { "reasoning": "`cfg` can be nil on line 42; dereferenced without check on line 47", "finding": "Possible nil‑pointer dereference", "confidence": 0.81 } ``` You know the confidence value is completely bogus, don't you?
The problem is that, regardless of how you try to use "micro-agents " as a marketing term, LLMs are instructed to return a result. They will always try to come up with something. The example provided was a poor one. The comment from LLM was solid. Why would you comment out a step in the pipeline instead of just deleting it? I would comment the same in a PR.
It takes less effort to re-enable if it's just commented out and its more visible that there is something funky going on that someone should fix.
But yeah, even if it's temporary, it really should have the rationale for commenting it out added... It takes like 5s and provides important context for reviewers and people looking through the file history in the future.
I agree with the sentiment of this post. I my personal experience the usefulness of a LLM positively correlated with your ability to constrain the problem it should solve. Prompts like 'Update this regex to match this new pattern' generally give better results than 'Fix this routing error in my server'. Although this pattern seems true empirically, I've never seen any hard data to confirm this property(?). And this p…
Funny thing is the structured output in the last example. ``` { "reasoning": "`cfg` can be nil on line 42; dereferenced without check on line 47", "finding": "Possible nil‑pointer dereference", "confidence": 0.81 } ``` You know the confidence value is completely bogus, don't you?
Do you mean that there is no correlation between confidence and false positives or other errors?
The multi agent thing with different roles is so obviously not a great concept, that I am very hesitant to build towards it, even thought it seems to win out right now. We want a AI that internally does what it needs to do to solve a problem, given a good enough problem description, tools and context. I really do not want to have to worry about breaking up tasks into chunks that are smaller than what I could handle m…
Funny thing is the structured output in the last example. ``` { "reasoning": "`cfg` can be nil on line 42; dereferenced without check on line 47", "finding": "Possible nil‑pointer dereference", "confidence": 0.81 } ``` You know the confidence value is completely bogus, don't you?
{
"reasoning": "`cfg` can be nil on line 42; dereferenced without check on line 47",
"finding": "Possible nil‑pointer dereference",
"confidence": 0.81,
"confidence_in_confidence_rating": 0.54,
"confidence_in_confidence_rating_in_confidence_rating": 0.12,
"confidence_in_confidence_rating_in_confidence_rating_in_confidence_rating": 0.98,
// Etc...
}I wonder what models they are using because reasoning models do this by default, even if they don't give you that output.
This post reads more like a marketing blog post than any real world advice.
Funny thing is the structured output in the last example. ``` { "reasoning": "`cfg` can be nil on line 42; dereferenced without check on line 47", "finding": "Possible nil‑pointer dereference", "confidence": 0.81 } ``` You know the confidence value is completely bogus, don't you?
Easy fix, just have the LLM generate: { "reasoning": "`cfg` can be nil on line 42; dereferenced without check on line 47", "finding": "Possible nil‑pointer dereference", "confidence": 0.81, "confidence_in_confidence_rating": 0.54, "confidence_in_confidence_rating_in_confidence_rating": 0.12, "confidence_in_confidence_rating_in_confidence_rating_in_confidence_rating": 0.98, // Etc... }
Funny thing is the structured output in the last example. ``` { "reasoning": "`cfg` can be nil on line 42; dereferenced without check on line 47", "finding": "Possible nil‑pointer dereference", "confidence": 0.81 } ``` You know the confidence value is completely bogus, don't you?