Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codex Security

github.com

Post reply on HN
Print view View original

81–90 of 257 posts

Re: Codex Security

#81

Post by dangelosaurus »

robotswantdata wrote: post #17
dangelosaurus wrote: post #13

Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI. Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly. If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here. CLI do…

Been watching your progress for a while, glad OpenAI have looked after you and the team and you still get to ship!

Thank you, that means a lot. Being able to keep building practical, open-source security tooling was important to us.

Really glad we got to ship this, and there's still a lot we want to improve in Codex Security and in Promptfoo!

Re: Codex Security

#83

Post by raylad »

petilon wrote: post #26

How does it work? Does the tool upload code to ChatGPT for analysis? That may not be allowed for some corporate projects.

Anything you ever do with any non-locally-hosted model always "uploads code" to the inference provider because that's how it works: the model uses tools to inspect the code, the result of the tool use is sent in an API call to provide context (and a prompt for the next turn), and then the response continues the process.

This is true and has to be true for any hosted model that works with existing code: it's not specific to this application.

Re: Codex Security

#84

Post by alansaber »

gregwebs wrote: post #41

Just ran it on a small repo. It ran for almost an hour and then got interrupted. It drained half my weekly usage on a Pro plan. npx codex-security scan . [00:00] Preparing scan [00:00] Authentication: stored Codex credentials. [00:03] Preparing scan [01:20] Running scan [01:20] Preflight: worker delegation supported (up to 8 worker slots). [52:47] Running scan codex-security: Could not save the Codex Security scan: R…

Classic.

Re: Codex Security

#86

Post by alansaber »

iancarroll wrote: post #33

Looks great but the CLI output is not particularly interesting while the scan is running. I wish it could show token usage, some kind of progress, etc.

Yes this is my pet peeve with a lot of the more involved agent skills/processes

Re: Codex Security

#87

Post by waterTanuki »

dangelosaurus wrote: post #13

Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI. Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly. If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here. CLI do…

Why does this need an entirely separate repo instead of being a feature in the existing Codex project?

Re: Codex Security

#89

Post by dangelosaurus »

varenc wrote: post #54

It's interesting how much of the value here is providing the english Skill definitions that tell the LLM what to do: https://github.com/openai/codex-security/tree/main/sdk/types... Some of approaches there could be useful in other contexts. OAI has the compute to experiment with different prompts and I'd expect these to be somewhat optimized.

Yes, I think this is an under-appreciated part of the release. I hope people can adapt them to their own workflows. We run A LOT of evals as the Promptfoo team and we've spent billions of tokens fine-tuning them. You can expect more skills as we branch out to other security workflows and further improvements to the codex security prompts.

Re: Codex Security

#90

Post by schrodinger »

Quick tangent if you’re willing to humor me…

I've been noticing that many new projects that would have been written in Python or Node a year ago are starting to be written in Go, Rust, etc.

Theory: people realized there’s little benefit to Python for agents. As Zep wrote, an “agent is a long-running, concurrent, I/O-bound process that spends most of its time waiting on a model, a tool, or a human[1]” — not a particular strength of Python.

I'm wondering if you'd considered Go (or others—Go’s just my fav ) before landing on Node, and more broadly whether you've noticed a similar pattern?

1: https://blog.getzep.com/agentic-development-in-go/

Post reply on HN