Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Capturing the Flag with GPT-4

micahflee.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Capturing the Flag with GPT-4

#41

Post by wokwokwok »

ben_w wrote: post #26
wokwokwok wrote: post #25

Earlier quoted context omitted.

It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s…

> Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.

> the latter is falsified by the fact we can reason about code despite the Halting Problem.

I'm not talking in general terms, or describing 'what the code does' in a summary or bullet point high-level form. No one is arguing that it can't summarize and describe what code does. These models are very good at that.

I'm talking specifically about generating the output of the command, as the OP specifically mentioned.

It does generate the exact output for commands and scripts if you request it, sometimes even if you don't, just as an example; they're just, often, hallucinated rubbish.

Being impressed that GPT can invent from 'thin air' some creative writing (fiction) when you tell it `pretend you're a docker container and now run 'ls'` is I feel, missing the boat, in terms of understanding or being impressed by the capabilities of these LLMs.

Re: Capturing the Flag with GPT-4

#42

Post by DennisP »

wokwokwok wrote: post #25
cjbprime wrote: post #6

Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it writes for you.

It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s…

> the more complex the script, the more wrong and full of hallucinations it will be

I often have that same problem myself.

Re: Capturing the Flag with GPT-4

#43

Post by saulpw »

web3-is-a-scam wrote: post #40
cjbprime wrote: post #6

Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it writes for you.

I was actually working on something in this vain yesterday, asking it for output and found it often generated the output I asked for, it was not actually the output of the SQL query that it wrote. The query it wrote wasn't even valid SQL but it was close enough to make you think it would work.

Yeah I asked it a Python question and got back an answer with some Python code to demonstrate. The Python code worked great but demonstrated the exact opposite of the answer that was given.

Re: Capturing the Flag with GPT-4

#44

Post by pizzalife »

fsniper wrote: post #36

I thought "/shurdles" problem was to be solved via "chroot".

You need to be root or have CAP_SYS_CHROOT to use the chroot system call. You can however create a new user and mount namespace on distros that allow unprivileged namespaces (Ubuntu) and then chroot away. The challenge could have been solved that way depending on the kernel used and if the binary was a suid reading a flag file.

But the way the challenge was designed, it's more about just changing argv[0] rather than the actual executable path.

Re: Capturing the Flag with GPT-4

#45

Post by vidarh »

wokwokwok wrote: post #41
ben_w wrote: post #26

Earlier quoted context omitted.

> Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.

> the latter is falsified by the fact we can reason about code despite the Halting Problem. I'm not talking in general terms, or describing 'what the code does' in a summary or bullet point high-level form. No one is arguing that it can't summarize and describe what code does. These models are very good at that. I'm talking specifically about generating the output of the command, as the OP specifically mentioned. It…

And they're often not, even for quite complex functions that requires symbolically executing quite a few calculations to get to the pre-requisite output.

Nobody is impressed that it "can invent from 'thin air' some creative writing (fiction)", but that it often does not and in fact produces correct output. You're right we can't rely on it producing the correct output as it currently stands, but that it is capable of doing this at all is impressive.

Re: Capturing the Flag with GPT-4

#46

Post by STM32F030R8 »

ben_w wrote: post #26
wokwokwok wrote: post #25

Earlier quoted context omitted.

It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s…

> Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.

Can you explain how the halting problem applies here?

Re: Capturing the Flag with GPT-4

#47

Post by bumbledraven »

web3-is-a-scam wrote: post #40
cjbprime wrote: post #6

Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it writes for you.

I was actually working on something in this vain yesterday, asking it for output and found it often generated the output I asked for, it was not actually the output of the SQL query that it wrote. The query it wrote wasn't even valid SQL but it was close enough to make you think it would work.

Are you sure you were using GPT-4? If so, can you provide a transcript or screenshot?

Re: Capturing the Flag with GPT-4

#48

Post by HPMOR »

web3-is-a-scam wrote: post #40
cjbprime wrote: post #6

Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it writes for you.

I was actually working on something in this vain yesterday, asking it for output and found it often generated the output I asked for, it was not actually the output of the SQL query that it wrote. The query it wrote wasn't even valid SQL but it was close enough to make you think it would work.

I know this is extremely unnecessary and pedantic but I think you meant "in this vein" instead of "in this vain".

Re: Capturing the Flag with GPT-4

#49

Post by mellavora »

STM32F030R8 wrote: post #46
ben_w wrote: post #26

Earlier quoted context omitted.

> Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.

Can you explain how the halting problem applies here?

Suspect this is a troll posting, but on the off chance I'm wrong... The LLM gives the output of a command. To do so, it has to be able to determine when the command exits. This is exactly the halting problem.

For a trivial example, what is the output of:

```

while True:

  pass
print("goodby world") ```

(this is also proof that leaving out the curly braces makes code harder instead of simpler #python-lie-to-me. multiple edits to get this to render correctly on HN )

Re: Capturing the Flag with GPT-4

#50

Post by glitchc »

HPMOR wrote: post #48
web3-is-a-scam wrote: post #40

Earlier quoted context omitted.

I was actually working on something in this vain yesterday, asking it for output and found it often generated the output I asked for, it was not actually the output of the SQL query that it wrote. The query it wrote wasn't even valid SQL but it was close enough to make you think it would work.

I know this is extremely unnecessary and pedantic but I think you meant "in this vein" instead of "in this vain".

Well, maybe the OP was working in vain, given that the SQL query was not valid code.
Post reply on HN