Live data from Hacker News

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

scalex.dev

151–160 of 268 posts

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#151

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

Back in my pen testing days, the best way to get someone to do something stupid and click Accept on something like a Java Applet permission check was to annoy them. Pop up a few annoying modals and eventually they just click yes on everything.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#152
post #94

Earlier quoted context omitted.

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases we…

Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces se…

> selfhosted gitlab - there was no option to reduce password complexity for my users

Gitlab's default requirements aren't that intense, but you can make them more stringent if you want.

https://docs.gitlab.com/user/profile/user_passwords/#passwor...

I guess your goal was to allow users to have 4 character passwords, i.e. "love." which afaik you can't do.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#153
post #115

Earlier quoted context omitted.

if firefly has no nulls, how do you indicate that a value is unset?

In Firefly (as in Rust) you can define fields as Optional, so you can do Option[String]; that lets you say "this variable is a String but it might not be here". That lets you then check to see if something is set, rather than checking to see if it's null. In Rust an Option is a separate thing that you need to disambiguate to use. For example: match result { // The division was valid Some(x) => println!("Result: {x}")…

oh duh! sorry, morning brain didn't think things through -- i do love Option types!

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#154

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

In my most recent job I was definitely under that sort of timer.

[deleted]

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#155
Yeah if you are trying to manually validate a firehouse of agent commands you are already losing before you started...

You sandbox, you have good checkpoints, and good agents, that's it. If you are manually reviewing commands you are wasting your time.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#156
post #74

Earlier quoted context omitted.

A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities. I've done some stabby s…

What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags. Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the langua…

If you want a fun read that might educate via entertaining I would read Satan Comes to Dinner by Douglas Crockford https://www.crockford.com/ec/dining.html

I wrote a bunch more, but I have a habit to verbosity when capabilities come in which I should attempt to combat, and have deleted it. Crockford says these things better than I can anyhow.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#157

Earlier quoted context omitted.

Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake. Just bringing it up because you're right, in software that's considered a bad pattern (rightfully so).

> Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake The "user[s] on both sides" of ATC conversions have passed through the filters of rigorous training and certification. They also happen to communicate in a DSL designed to minimize misunderstandings, the DSL just happens to be based on English.

I don't think OPs argument here was simply that the users aren't qualified enough to approve llm output.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#158
post #121

Earlier quoted context omitted.

I think you're confused. The verb form of the word never carried the credentialism of the title. In the same way that "doctoring" never carried the connotation of a medical degree. Of course the original sense of the noun was "a person who devises things" and shares a root with "ingenious" and carried no connotation of legal credential. That "harm" is more or less restorative to the original meaning of the word.

I'm not referring to the verb form, I'm referring to the people who call themselves things like "prompt engineer" or "software engineer" with a straight face, draping themselves in a false legitimacy stolen from professionals for whom the term "engineer" actually implied something of note. It's embarrassing, or it would be if people were still possessed of the capacity for shame.

> I'm not referring to the verb form

The original post prompting your complaint used the verb form. And hence why you're confused.

> It's embarrassing, or it would be if people were still possessed of the capacity for shame.

It's far more shameful to fly off the cuff with a non-sequitur by your own admission, and double down on yelling at the clouds when called out for it. And about an arbitrary point in a semantic treadmill, no less. Your take away is that it was never something of note for these people, and trying to pick semantic fights with them over it just makes crying about legitimacy ironically turn into pure pretense.

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#159
post #26

Earlier quoted context omitted.

Yet the AI can still escape the "sandbox", unless it is physically unable to connect to another computer and completely airgapped.

If the sandbox has vulnerabilities, which you can also use the AI to fuzz for. Obviously at the point in which it can talk to the internet it doesn't really matter, but there are a very finite number of zero-days that can exist in a bytecode interpreter hosting a harness.

Well it turns out that we have yet another sandbox escape just released today called "Zapscape".

My point is if an agent recited how to find one in its memory or training set and it is air-gapped, the chances of it spreading and infecting other computers is pretty low.

[0] https://news.ycombinator.com/item?id=49198843

Re: Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

#160
post #44

I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless. Also the game was on a timer, and maybe ther…

I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandbox…

For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.
Post reply on HN