21–30 of 79 posts
The big disconnect here is I can't share code with OpenAI for various reasons.
Would you consider using something like AWS Bedrock+Anthropic Claude, where we have better (more predictable/risk profile) control over data-sharing, etc?
I know you can't take responsibility for the 5% failure rate GPT-4 produces, but maybe things change when you have your simulator running. At that point, what kind of SLA do you plan on offering with the service?
More important than SLAs is who takes on the liability for these mistakes? With the changing laws and regulations for breaches, do I want to rely on an LLM that isn't going to own that liability?
Again, I understand the skepticism using LLM's but currently everything is done manually and it shows that doesn't work well. So using LLM's is a quick way to improve the current situation and hopefully we can further compliment it with checks and balances
A challenge with Slauth will be to organize the generated policies in a way that makes them legible. I would like the IAM policy to help clarify its intent. Allowing each in-use API endpoint is technically required to let the service work. It might be technically following the principal of least privilege. But the endpoint-by-endpoint rules do a poor job of summarizing the purpose of the policy or how it relates to the service. One way that we do this is by having resource providers declare managed policies that allow use of the resource. So the "encabulator" provider also defines a "mutate encabulator" managed policy. Then services that need to invoke "mutate encabulator" can reference the managed policy. They don't need to compute the ARN of every endpoint. The dependent service doesn't end up with an inline policy that has detailed dependencies on the implementation details of each target resource.
Repost: https://news.ycombinator.com/item?id=34038663 (11 months ago) > We employ Large Language Models (currently OpenAI GPT-4) For IAM, this seems like a disaster waiting to happen. Combining hallucination problems with security settings is not a path I would consider
Do you think humans are doing a better job? Research shows that 95% of the permissions granted to users aren't used which creates huge problems and is a reason for spending millions in security tools. Why not use Slauth and other checks such as policy simulators to get tightened policies pre-deployed
On another note, are there tools that will scan your AWS/GCP logs and emit configuration to limit permissions to what you're actually using? I could even see GPT doing better here too, or at least it would be easier to test.
Repost: https://news.ycombinator.com/item?id=34038663 (11 months ago) > We employ Large Language Models (currently OpenAI GPT-4) For IAM, this seems like a disaster waiting to happen. Combining hallucination problems with security settings is not a path I would consider
Do you think humans are doing a better job? Research shows that 95% of the permissions granted to users aren't used which creates huge problems and is a reason for spending millions in security tools. Why not use Slauth and other checks such as policy simulators to get tightened policies pre-deployed
These would be the "s3:*" and "Resources: *" scoped permissions I assume? I can't imagine users are explicitly typing out permissions, 95% of which are not relevant for the task.
> which creates huge problems
Such as? What is the material impact of a workflow or a user having too many permissions?
> and is a reason for spending millions in security tools
Are you claiming that overscoped IAM permissions alone are responsible for 1M+ security tooling bills in companies? Would you be willing to share information on which tools these are?
Earlier quoted context omitted.
More important than SLAs is who takes on the liability for these mistakes? With the changing laws and regulations for breaches, do I want to rely on an LLM that isn't going to own that liability?
Perhaps I should have emphasized better that indeed the LLM's are trustworthy by themselves and require several extra checks. These would be policy simulators, connecting to cloud environments and running checks in Dev/Staging. Again, I understand the skepticism using LLM's but currently everything is done manually and it shows that doesn't work well. So using LLM's is a quick way to improve the current situation and…
If it is all done manually, and there are both good and bad IAM setups, can you really extrapolate to "manual" being the root cause? How can you even get an LLM to produce secure policies without having existing secure policies to train on? The entire premiss seems off and misleading to me
I would expect a hands-off approach to have worse outcomes
Repost: https://news.ycombinator.com/item?id=34038663 (11 months ago) > We employ Large Language Models (currently OpenAI GPT-4) For IAM, this seems like a disaster waiting to happen. Combining hallucination problems with security settings is not a path I would consider
Do you think humans are doing a better job? Research shows that 95% of the permissions granted to users aren't used which creates huge problems and is a reason for spending millions in security tools. Why not use Slauth and other checks such as policy simulators to get tightened policies pre-deployed