[dead]
Capturing the Flag with GPT-4
51–60 of 60 posts
Re: Capturing the Flag with GPT-4
#52Earlier quoted context omitted.
I was actually working on something in this vain yesterday, asking it for output and found it often generated the output I asked for, it was not actually the output of the SQL query that it wrote. The query it wrote wasn't even valid SQL but it was close enough to make you think it would work.
I know this is extremely unnecessary and pedantic but I think you meant "in this vein" instead of "in this vain".
yes, thank you I always get them mixed up
Re: Capturing the Flag with GPT-4
#53Earlier quoted context omitted.
Can you explain how the halting problem applies here?
Suspect this is a troll posting, but on the off chance I'm wrong... The LLM gives the output of a command. To do so, it has to be able to determine when the command exits. This is exactly the halting problem. For a trivial example, what is the output of: ``` while True: pass print("goodby world") ``` (this is also proof that leaving out the curly braces makes code harder instead of simpler #python-lie-to-me. multiple…
Rice's theorem may actually be more appropriate here (but it's a consequence of the halting theorem).
But it's important to note that just because there's no algorithm that works on ALL programs doesn't mean that the semantic properties of all programs are undecidable. Clearly for the particular programs where the program is bounded and guaranteed to terminate (e.g. no unbounded loops or recursion allowed) we can determine such properties, and I believe theorem provers in fact only allow such programs. And similarly you can restrict yourself to only the programs that you can prove will terminate in N steps (which might be excluding some programs that do terminate but require more than N steps of compute to prove).
Re: Capturing the Flag with GPT-4
#54I thought "/shurdles" problem was to be solved via "chroot".
You need to be root or have CAP_SYS_CHROOT to use the chroot system call. You can however create a new user and mount namespace on distros that allow unprivileged namespaces (Ubuntu) and then chroot away. The challenge could have been solved that way depending on the kernel used and if the binary was a suid reading a flag file. But the way the challenge was designed, it's more about just changing argv[0] rather than…
Yep, challenge author here, and it was definitely to teach that `argv[0]` is not trustworthy. I've seen privileged processes try to re-invoke themselves (as, say, a child process) by looking at `argv[0]` rather than something like `/proc/self/exe` (which is also subject to race conditions if the directory is writable).
The binary was not setuid, but was only executable (not readable) by the user used.
Re: Capturing the Flag with GPT-4
#55Earlier quoted context omitted.
I've run CTF games at major conferences. The point is to solve the challenges by any means necessary within the rules... and that which is not forbidden is allowed. If I, as a person running a CTF, did not want my players to do this, I would set up a few problems which would have incorrect (but not obviously so) "solutions" generated when fed to LLM. The Shamir's Secret Sharing reminds me of the time I was playing DE…
> it involved hiding fragments of a split secret in a modified version of ADVENT. I solved it. Even when the board was fully opened, it was nowhere to be seen Can you explain what this means? I don't understand, except for the split secret part. Also: How did you do it?
https://en.wikipedia.org/wiki/Colossal_Cave_Adventure
http://point-at-infinity.org/ssss/
DEF CON CTF quals is (or was) "Jeopardy" style with five categories with five problems each. The thing I found was not one of the 25 problems.
Re: Capturing the Flag with GPT-4
#56Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it writes for you.
It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s…
Obviously I wouldn't have been impressed by hallucinated output. I'm talking about correctly modeling a variable's state through its conception of what a Python interpreter does, which requires building a model of that interpreter and extracting consequences from it, rather than pure language statistics.
I agree that LLMs will often hallucinate. There is obviously no guarantee that the output is correct. But sometimes it is correct anyway, which I notice by actually running the code.
Here is a trivial example which I only mention to bring the conversation back to the reality of GPT-4 actually being able to do things like this:
Me:
You are a Python interpreter. Please give the correct output of the supplied code, with no commentary.
>>> a = ["wokwokwok", "says", "i", "have", "no", "understanding", "of", "code"]
>>> a.append("!")
>>> " ".join([w.upper() for w in a])
WOKWOKWOK SAYS I HAVE NO UNDERSTANDING OF CODE !
Re: Capturing the Flag with GPT-4
#57I thought "/shurdles" problem was to be solved via "chroot".
You need to be root or have CAP_SYS_CHROOT to use the chroot system call. You can however create a new user and mount namespace on distros that allow unprivileged namespaces (Ubuntu) and then chroot away. The challenge could have been solved that way depending on the kernel used and if the binary was a suid reading a flag file. But the way the challenge was designed, it's more about just changing argv[0] rather than…
Yes. Thank you. I just missed that part.
It's been a long while since doing basic linux administration. I am getting rusty.
Re: Capturing the Flag with GPT-4
#58Earlier quoted context omitted.
You need to be root or have CAP_SYS_CHROOT to use the chroot system call. You can however create a new user and mount namespace on distros that allow unprivileged namespaces (Ubuntu) and then chroot away. The challenge could have been solved that way depending on the kernel used and if the binary was a suid reading a flag file. But the way the challenge was designed, it's more about just changing argv[0] rather than…
Yep, challenge author here, and it was definitely to teach that `argv[0]` is not trustworthy. I've seen privileged processes try to re-invoke themselves (as, say, a child process) by looking at `argv[0]` rather than something like `/proc/self/exe` (which is also subject to race conditions if the directory is writable). The binary was not setuid, but was only executable (not readable) by the user used.
>The binary was not setuid, but was only executable (not readable) by the user used.
Ah, then ptrace/gdb could have been used to dump it out as well :). Looks like a fun CTF, too bad I was too busy for bsides this year..
Re: Capturing the Flag with GPT-4
#59Earlier quoted context omitted.
It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s…
Obviously I wouldn't have been impressed by hallucinated output. I'm talking about correctly modeling a variable's state through its conception of what a Python interpreter does, which requires building a model of that interpreter and extracting consequences from it, rather than pure language statistics. I agree that LLMs will often hallucinate. There is obviously no guarantee that the output is correct. But sometime…
That’s because you’ve asked for something that is trivially derived.
> Tldr; yes, but it doesn’t scale well beyond trivial outputs.
Re: Capturing the Flag with GPT-4
#60Earlier quoted context omitted.
Obviously I wouldn't have been impressed by hallucinated output. I'm talking about correctly modeling a variable's state through its conception of what a Python interpreter does, which requires building a model of that interpreter and extracting consequences from it, rather than pure language statistics. I agree that LLMs will often hallucinate. There is obviously no guarantee that the output is correct. But sometime…
That’s because you’ve asked for something that is trivially derived. > Tldr; yes, but it doesn’t scale well beyond trivial outputs.
Which algorithms are trivial and which are non-trivial? Is sorting trivial? base64-encoding? Can you provide a set of small scripts that you expect GPT-4 to be unable to correctly simulate execution of?
It is weird that you seem to agree that it is capable of performing algorithmic simulation, while discounting that with "but it’s not executing the code; it’s generating what seems like plausible output", in a way that seems suspiciously close to defining anything it simulates correctly as "trivial", and anything it would fail at as "executing the code"...