421–430 of 569 posts
Does anyone know a good alternate project that works similarly (share multipple LLMs across a set of users)? LiteLLM has been getting worse and trying to get me to upgrade to a paid version. I also had issues with creating tokens for other users etc.
We just can't trust dependencies and dev setups. I wanted to say "anymore" but we never could. Dev containers were never good enough, too clumsy and too little isolation. We need to start working in full sandboxes with defence in depth that have real guardrails and UIs like vm isolation + container primitives and allow lists, egress filters, seccomp, gvisor and more but with much better usability. Its the same requir…
Happily sandboxing almost all third-party tools since 2025. `npm run dev` does not need access to my full disk.
Earlier quoted context omitted.
This is the security shortcuts of the past 50 years coming back to bite us. Software has historically been a world where we all just trust each other. I think that’s coming to an end very soon. We need sandboxing for sure, but it’s much bigger than that. Entire security models need to be rethought.
This assumes that we can get a locked down, secure, stable bedrock system and sandbox that basically never changes except for tiny security updates that can be carefully inspected by many independent parties. Which sounds great, but the way things work now tend to be the exact opposite of that, so there will be no trustable platform to run the untrusted code in. If the sandbox, or the operating system the sandbox run…
Not really. You should limit the attack surface for third-party code.
A linter running in `dir1` should not access anything outside `dir1`.
What is the best way to sandbox LLMs and packages in general, while being able to work on data from outside sandbox (get data in and out easily)? There is also the need for data sanitation, because the attacker could distribute compromised files through user’s data which will later be run and compromise the host.
Earlier quoted context omitted.
I think we really need to use sandboxes. Guix provides sandboxed environments by just flipping a switch. NixOS is in an ideal position to do the same, but for some reason they are regarded as "inconvenient". Personally, I am a heavy user of Firejail and bwrap. We need defense in depth. If someone in the supply chain gets compromised, damage should be limited. It's easy to patch the security model of Linux with usersp…
What would be really helpful is if software sandboxed itself. It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive. But, sadly, there's no x-platform way to do this, and sandboxing APIs are incredibly bad still and often require privileges. > It's easy to patch the security model of Linux with userspaces, and even easier with eBPF…
(And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)
Earlier quoted context omitted.
What would be really helpful is if software sandboxed itself. It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive. But, sadly, there's no x-platform way to do this, and sandboxing APIs are incredibly bad still and often require privileges. > It's easy to patch the security model of Linux with userspaces, and even easier with eBPF…
> It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive. Not really. Let's say I am running `~/src/project1 $ litellm` Why does this need access to anything outside of `~/src/project1`? Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.
> Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.
Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.
Earlier quoted context omitted.
>1. Looks like this originated from the trivvy used in our ci/cd Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?
The latest trivy attack was announced just yesterday. If you go out to dinner or take a night off its totally plausible to have not seen it.
Earlier quoted context omitted.
we're using litellm via helm charts with tags main-v1.81.12-stable.2 and main-v1.80.8-stable.1 - assuming they're safe? also how are we sure that docker images aren't affected?
Docker deployments are more safe even if affected because there is a lower chance (but not zero) that you didn't mount all your credentials into the image. It would have access to LLM keys of course, but that's not really what the hacker is after. He's after private SSH keys. That being said this hack was a direct upload to PyPI in the last few days, so very unlikely those images are affected.
https://github.com/dweinstein/canary I made this tool for macos systems that helps detect when a package accesses something it shouldn't. it's a tiny go binary (less than 2k LOC) with no dependencies that will mount a webdav filesystem (no root) or NFS (root required) with fake secrets and send you a notification when anything accesses it. Very stupid simple. I've always really liked the canary/honeypot approach and…
Earlier quoted context omitted.
This is the security shortcuts of the past 50 years coming back to bite us. Software has historically been a world where we all just trust each other. I think that’s coming to an end very soon. We need sandboxing for sure, but it’s much bigger than that. Entire security models need to be rethought.
This assumes that we can get a locked down, secure, stable bedrock system and sandbox that basically never changes except for tiny security updates that can be carefully inspected by many independent parties. Which sounds great, but the way things work now tend to be the exact opposite of that, so there will be no trustable platform to run the untrusted code in. If the sandbox, or the operating system the sandbox run…