Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

github.com

Post reply on HN
Print view View original

311–320 of 569 posts

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#311

Post by saltyoldman »

kingreflex wrote: post #296
detente18 wrote: post #99

LiteLLM maintainer here, this is still an evolving situation, but here's what we know so far: 1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09 2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt 3. The package is in quarantine on pypi - this block…

we're using litellm via helm charts with tags main-v1.81.12-stable.2 and main-v1.80.8-stable.1 - assuming they're safe? also how are we sure that docker images aren't affected?

Docker deployments are more safe even if affected because there is a lower chance (but not zero) that you didn't mount all your credentials into the image. It would have access to LLM keys of course, but that's not really what the hacker is after. He's after private SSH keys.

That being said this hack was a direct upload to PyPI in the last few days, so very unlikely those images are affected.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#312

Post by heavyset_go »

dotancohen wrote: post #286
adw wrote: post #177

Earlier quoted context omitted.

> In one of my vibe coded personal projects (Python and Rust project) I'm actually getting rid of most dependencies and vibe coding replacements that do just what I need. I think that we'll see far fewer dependencies in future projects. No free lunch. LLMs are capable of writing exploitable code and you don’t get notifications (in the eg Dependabot sense, though it has its own problems) without audits.

My vibe coded personal projects don't have the source code available for attackers to target specifically.

You don't need open source access to be exploitable or exploited

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#313

Post by [deleted] »

tedivm wrote: post #283
mike_hearn wrote: post #248

Earlier quoted context omitted.

Perhaps it's too obvious but ... just running the publish process locally, instead of from CI, would help. Especially if you publish from a dedicated user on a Mac where the system keychain is pretty secure.

This problem is solved by not having a token. Github and PyPI both support OIDC based workflows. Grant only the publish job access to OIDC endpoint, then the Trivy job has nothing it can steal.

[deleted]

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#314

Post by ramimac »

miraculixx wrote: post #301
ramimac wrote: post #42

This is tied to the TeamPCP activity over the last few weeks. I've been responding, and keeping an up to date timeline. I hope it might help folks catch up and contextualize this incident: https://ramimac.me/trivy-teampcp/#phase-09

This is interesting. How do you keep this up to date so quickly?

Blood, sweat, and tears.

The investment compounds! I have enough context to quickly vet incoming information, then it's trivial to update a static site with a new blurb

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#315

Post by ting0 »

cedws wrote: post #129
jFriedensreich wrote: post #86

We just can't trust dependencies and dev setups. I wanted to say "anymore" but we never could. Dev containers were never good enough, too clumsy and too little isolation. We need to start working in full sandboxes with defence in depth that have real guardrails and UIs like vm isolation + container primitives and allow lists, egress filters, seccomp, gvisor and more but with much better usability. Its the same requir…

This is the security shortcuts of the past 50 years coming back to bite us. Software has historically been a world where we all just trust each other. I think that’s coming to an end very soon. We need sandboxing for sure, but it’s much bigger than that. Entire security models need to be rethought.

What we need is accountability and ties to real-world identity.

If you're compromised, you're burned forever in the ledger. It's the only way a trust model can work.

The threat of being forever tainted is enough to make people more cautious, and attackers will have no way to pull off attacks unless they steal identities of powerful nodes.

Like, it shouldn't be a thing that some large open-source project has some 4th layer nested dependency made by some anonymous developer with 10 stars on Github.

If instead, the dependency chain had to be tied to real verified actors, you know there's something at stake for them to be malicious. It makes attacks much less likely. There's repercussions, reputation damage, etc.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#316

Post by ting0 »

klibertp wrote: post #226
cedws wrote: post #129

Earlier quoted context omitted.

This is the security shortcuts of the past 50 years coming back to bite us. Software has historically been a world where we all just trust each other. I think that’s coming to an end very soon. We need sandboxing for sure, but it’s much bigger than that. Entire security models need to be rethought.

The NIH syndrome becoming best practice (a commenter below already says they "vibe-coded replacements for many dependencies") would also save quite a few jobs, I suspect. Fun times.

I've been doing that too. The downside is it's a lot of work for big replacements.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#317

Post by snailmailman »

hiciu wrote: post #15

Besides main issue here, and the owners account being possibly compromised as well, there's like 170+ low quality spam comments in there. I would expect better spam detection system from GitHub. This is hardly acceptable.

The same thing occurred on the trivy repo a few days ago. A GitHub discussion about the hack was closed and 700+ spam comments were posted.

I scrolled through and clicked a few profiles. While many might be spam accounts or low-activity accounts, some appeared to be actual GitHub users with a history of contributions.

I’m curious how so many accounts got compromised. Are those past hacks, or is this credential steeling hack very widespread?

Are the trivy and litellm hacks just 2 high profile repos out of a much more widespread “infect as many devs as possible, someone might control a valuable GitHub repository” hack? I’m concerned that this is only the start of many supply chain issues.

Edit: Looking through and several of the accounts have a recent commit "Update workflow configuration" where they are placing a credential stealer into a CI workflow. The commits are all back in february.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#318

Post by itintheory »

ramimac wrote: post #42

This is tied to the TeamPCP activity over the last few weeks. I've been responding, and keeping an up to date timeline. I hope it might help folks catch up and contextualize this incident: https://ramimac.me/trivy-teampcp/#phase-09

Thanks for putting this together. I've been seeing the name TeamPCP pop up all over, but hadn't seen everything in one place.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#319

Post by mike_hearn »

staticassertion wrote: post #252
mike_hearn wrote: post #248

Earlier quoted context omitted.

Perhaps it's too obvious but ... just running the publish process locally, instead of from CI, would help. Especially if you publish from a dedicated user on a Mac where the system keychain is pretty secure.

I'm not sure how. Their local system seems just as likely to get compromised through a `pip install` or whatever else. In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.

User isolation works, the keychain isolation works. On macOS tokens stored in the keychain can be made readable only by specific apps, not anything else. It does require a bit of infrastructure - ideally a Mac app that does the release - but nothing you can't vibe code quickly.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#320

Post by staticassertion »

mike_hearn wrote: post #319
staticassertion wrote: post #252

Earlier quoted context omitted.

I'm not sure how. Their local system seems just as likely to get compromised through a `pip install` or whatever else. In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.

User isolation works, the keychain isolation works. On macOS tokens stored in the keychain can be made readable only by specific apps, not anything else. It does require a bit of infrastructure - ideally a Mac app that does the release - but nothing you can't vibe code quickly.

That's true, but it seems far more complex than just moving trivy to a separate workerflow with no permissions and likely physical isolation between it and a credential. I'm pretty wary of the idea that malware couldn't just privesc - it's pretty trivial to obtain root on a user's laptop. Running as a separate, unprivileged user helps a ton, but again, I'm skeptical of this vs just using a github workflow.
Post reply on HN