Perhaps this article was written by the same AI that failed to understand what it was supposed to do in the first place? The post doesn't make a lot of sense and the writing seems fishy. I still don't understand what was wrong with he first code.
A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
11–18 of 18 posts
Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
#12I often find myself clearing the context when dealing with llms to get a fresh take. Often it just has so much context reinforcing its previous decisions.
Not sure if the author tried to just start a new thread. But anyway, for now you always need to keep an eye on these things and manage it if it follows red herrings or ends up in some logical loop
Sidenote : newlines is one thing tat can be quite tricky for llms in general.
Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
#13Not exactly the point of this article, but it would be cool if APIs like this can return the expected signed string for debugging. It would have to be properly limited for security. But if the API is expecting non-standard signatures, it could help developers with better debugging tools.
Given that you can't infer the error from simply looking at the signature string, I don't see how having the expected string rather than a simple "OK" or "mismatched signature" (as you get now) would make a difference?
You can save the expected string to a file, save your string to a file, and run diff on a hexdump of both. Even without hexdump, you should see the difference between "\n" and "\\n" in properly escaped output.
Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
#14Earlier quoted context omitted.
Given that you can't infer the error from simply looking at the signature string, I don't see how having the expected string rather than a simple "OK" or "mismatched signature" (as you get now) would make a difference?
You can save the expected string to a file, save your string to a file, and run diff on a hexdump of both. Even without hexdump, you should see the difference between "\n" and "\\n" in properly escaped output.
But the returned signed string will be an HMAC-SHA256 hash, won't it? Then there's not going to be any '\n' or '\\n's in there. Only thing you'll be able to tell is if it matches your hash or not, in which case 'OK' or 'not OK' will work just as well.
Or am I misunderstanding you?
Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
#15Earlier quoted context omitted.
You can save the expected string to a file, save your string to a file, and run diff on a hexdump of both. Even without hexdump, you should see the difference between "\n" and "\\n" in properly escaped output.
But the returned signed string will be an HMAC-SHA256 hash, won't it? Then there's not going to be any '\n' or '\\n's in there. Only thing you'll be able to tell is if it matches your hash or not, in which case 'OK' or 'not OK' will work just as well. Or am I misunderstanding you?
You are indeed misunderstanding me. I am talking about returning the entire string to be signed. Not the result of the signature.
Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
#16Earlier quoted context omitted.
But the returned signed string will be an HMAC-SHA256 hash, won't it? Then there's not going to be any '\n' or '\\n's in there. Only thing you'll be able to tell is if it matches your hash or not, in which case 'OK' or 'not OK' will work just as well. Or am I misunderstanding you?
You are indeed misunderstanding me. I am talking about returning the entire string to be signed. Not the result of the signature.
Ah, my bad. Sorry.
But couldn't you then just make the call to an echo service (like HTTPbin) or simply dump the request when you send it?
Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
#17Earlier quoted context omitted.
You are indeed misunderstanding me. I am talking about returning the entire string to be signed. Not the result of the signature.
Ah, my bad. Sorry. But couldn't you then just make the call to an echo service (like HTTPbin) or simply dump the request when you send it?
The echo server will have no knowledge on how to construct the string to be signed.
Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs
#18Earlier quoted context omitted.
Ah, my bad. Sorry. But couldn't you then just make the call to an echo service (like HTTPbin) or simply dump the request when you send it?
The echo server will have no knowledge on how to construct the string to be signed.
But neither does the actual server. HMAC only verifies that the message is from whoever it claims to be from and that it is intact. It won't know what you intended the body of the request to look like.