Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Tale of Two AI Failures: Debugging a Simple Bug with LLMs

bitmovin.com

Post reply on HN
Print view View original

1–10 of 18 posts

Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs

#2

Post by kichik »

Not exactly the point of this article, but it would be cool if APIs like this can return the expected signed string for debugging. It would have to be properly limited for security. But if the API is expecting non-standard signatures, it could help developers with better debugging tools.

Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs

#3

Post by lillesvin »

I know it's kinda besides the point and I don't know what language this was being done in, but I don't personally know any language where

    String signature = "POST" + "\n" + "/api/v1/..."
and

    String signature = "POST\n/api/v1/..."
don't result in identical variables, so I'm a bit puzzled why that would result in an error.

However, there's a quoting error in the failing example where the double quotes in the JSON body aren't properly escaped:

    String signature = "POST" + "\n" + "/api/v1/query" + "\n" + token + "\n" + timestamp + "\n" + "{"body":"content"}"
It may just be the example that's not correctly formatted, but the other (working) example does in fact escape the double quotes in the JSON. I guess, depending on how forgiving the used language is with quoting, that could also be the source of the error?

Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs

#5

Post by lillesvin »

kichik wrote: post #2

Not exactly the point of this article, but it would be cool if APIs like this can return the expected signed string for debugging. It would have to be properly limited for security. But if the API is expecting non-standard signatures, it could help developers with better debugging tools.

Given that you can't infer the error from simply looking at the signature string, I don't see how having the expected string rather than a simple "OK" or "mismatched signature" (as you get now) would make a difference?

Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs

#7

Post by juancn »

lillesvin wrote: post #3

I know it's kinda besides the point and I don't know what language this was being done in, but I don't personally know any language where String signature = "POST" + "\n" + "/api/v1/..." and String signature = "POST\n/api/v1/..." don't result in identical variables, so I'm a bit puzzled why that would result in an error. However, there's a quoting error in the failing example where the double quotes in the JSON body…

Yeah, I'm stuck here.

Another thing that's really broken is the last string with unescaped quotes.

Not sure how to interpret that unless theres a `:` (colon) operator.

Re: A Tale of Two AI Failures: Debugging a Simple Bug with LLMs

#9

Post by saint_yossarian »

lillesvin wrote: post #3

I know it's kinda besides the point and I don't know what language this was being done in, but I don't personally know any language where String signature = "POST" + "\n" + "/api/v1/..." and String signature = "POST\n/api/v1/..." don't result in identical variables, so I'm a bit puzzled why that would result in an error. However, there's a quoting error in the failing example where the double quotes in the JSON body…

The author mentions FoxESSCloud, which led me to https://www.foxesscloud.com/public/i18n/en/OpenApiDocument.h... with this Python example:

    signature = fr'{path}\r\n{token}\r\n{timestamp}'
So if this is indeed the API they're using it's not only literal "\\n" but also "\\r\\n", no "POST", and no body at the end.
Post reply on HN