Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

github.com

Post reply on HN
Print view View original

501–510 of 569 posts

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#501

Post by azornathogron »

nazcan wrote: post #164
uyzstvqs wrote: post #153

Earlier quoted context omitted.

That's no solution. If you can't trust and/or verify dependencies, and they are malicious, then you have bigger problems than what a sandbox will protect against. Even if it's sandboxed and your host machine is safe, you're presumably still going to use that malicious code in production.

I'm supportive of going further - like restricting what a library is able to do. e.g. if you are using some library to compute a hash, it should not make network calls. Without sub-processes, it would require OS support.

In type system theory I think what you're looking for is "effect systems".

You make the type system statically encode categories of side-effects, so you can tell from the type of a function whether it is pure computation, or if not what other things it might do. Exactly what categories of side-effect are visible this way depends on the type system; some are more expressive than others.

But it means when you use a hash function you can know that it's, eg, only reading memory you gave it access to and doing some pure computation on it.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#502

Post by otabdeveloper4 »

garthk wrote: post #360
KoftaBob wrote: post #285

Earlier quoted context omitted.

What in the world are “the dangerous insights”?

“Society is a construct”, for starters?

That's babby's first insight. Most people figure this out on their own in kindergarten.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#503

Post by tomtomtom777 »

postalcoder wrote: post #453

FYI, npm/bun/pnpm/uv now all support setting a minimum release age for packages. I updated my global configs to set min release age to 7 days: ~/.config/uv/uv.toml exclude-newer = "7 days" ~/.npmrc min-release-age=7 # days ~/Library/Preferences/pnpm/rc minimum-release-age=10080 # minutes ~/.bunfig.toml [install] minimumReleaseAge = 604800 # seconds

I understand that this is a good idea but it does feel really weird. Add a min-release-age to see if anyone who doesn't gets bitten.

Next up, we're going to advise a minimum-release-age of 14 days, cause most other projects use 7 days.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#504

Post by BoppreH »

vlovich123 wrote: post #500
BoppreH wrote: post #494

Earlier quoted context omitted.

The proposed solution seems to rely on a trusted compiler that generates the exact same output, bit-for-bit, as the compiler-under-test would generate if it was not compromised. That seems useful only in very narrow cases.

You have a trusted compiler you write in assembly or even machine code. You then compile a source code you trust using that compiler. That is then used for the bit for bit analysis against a different binary of the compiler you produced to catch the hidden vulnerability.

It's assumed that in this scenario you don't have access to a trusted compiler; if you do, then there's no problem.

And the thesis linked above seems to go beyond simply "use a trusted compiler to compile the next compiler". It involves deterministic compilation and comparing outputs, for example.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#505

Post by ctmnt »

ctmnt wrote: post #483
davidatbu wrote: post #474

Earlier quoted context omitted.

I don't think pypi or npm allow replacing existing packages?

They absolutely do. In this case litellm 1.82.8 had been out for at least a week (can’t recall the exact date offhand). The compromised version was a replacement.

Ah, my mistake! Thanks for the correction.

But I believe you can replace versions on both, nonetheless. It’s a multi step process, unpublish then publish again. But the net effect is the same.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#506

Post by sobellian »

driftnode wrote: post #446
detente18 wrote: post #99

LiteLLM maintainer here, this is still an evolving situation, but here's what we know so far: 1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09 2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt 3. The package is in quarantine on pypi - this block…

the chain here is wild. trivy gets compromised, that gives access to your ci, ci has the pypi publish token, now 97 million monthly downloads are poisoned. was the pypi token scoped to publishing only or did it have broader access? because the github account takeover suggests something wider leaked than just the publish credential

If the payload is a credential stealer then they can use that to escalate into basically anything right?

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#507

Post by Bullhorn9268 »

I am from futuresearch and went through this with Callum (the OG). We did a small analysis here: https://futuresearch.ai/blog/litellm-hack-were-you-one-of-th... of the packages and also build this mini tool to analyze the likelihood of you getting pwned through this: https://futuresearch.ai/tools/litellm-checker/

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#508

Post by driftnode »

sobellian wrote: post #506
driftnode wrote: post #446

Earlier quoted context omitted.

the chain here is wild. trivy gets compromised, that gives access to your ci, ci has the pypi publish token, now 97 million monthly downloads are poisoned. was the pypi token scoped to publishing only or did it have broader access? because the github account takeover suggests something wider leaked than just the publish credential

If the payload is a credential stealer then they can use that to escalate into basically anything right?

Yes and the scary part is you might never know the full extent. A credential stealer grabs whatever is in memory or env during the build, ships it out, and the attacker uses those creds weeks later from a completely different IP. The compromised package gets caught and reverted, everyone thinks the incident is over, meanwhile the stolen tokens are still valid. I wonder how many teams who installed 1.82.7 actually rotated all their CI secrets after this, not just uninstalled the bad version.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#509

Post by driftnode »

kreelman wrote: post #476
driftnode wrote: post #446

Earlier quoted context omitted.

the chain here is wild. trivy gets compromised, that gives access to your ci, ci has the pypi publish token, now 97 million monthly downloads are poisoned. was the pypi token scoped to publishing only or did it have broader access? because the github account takeover suggests something wider leaked than just the publish credential

I wonder if there are a few things here.... It would be great if Linux was able to do simple chroot jails and run tests inside of them before releasing software. In this case, it looks like the whole build process would need to be done in the jail. Tools like lxroot might do enough of what chroot on BSD does. It seems like software tests need to have a class of test that checks whether any of the components of an app…

The sad part is you're right that we can't assume secure operation of components anymore, but the tooling hasn't caught up to that reality. Chroot jails help with runtime isolation but the attack here happened at build time, the malicious code was already in the package before any test could run. And the supply chain is deep. Trivy gets compromised, which gives CI access, which gives PyPI access. Even if you jail your own builds you're trusting that every tool in your pipeline wasn't the entry point. 97 million monthly downloads means a lot of people's "secure" pipelines just ran attacker code with full access.

Re: Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

#510

Post by pabs3 »

dist-epoch wrote: post #488
pabs3 wrote: post #434

Earlier quoted context omitted.

> Are you really checking all the files in there, even the binaries? One should never trust the binaries, always build them from source, all the way down to the bootloader. https://bootstrappable.org/ Checking all the files is really the only way to deal with potential malware, or even security vulns. https://github.com/crev-dev/

Nice ideal, but Chrome/Firefox would take days to build on your average laptop (if it doesn't run out of memory first).

The latest Firefox build that Debian did only took just over one hour on amd64/armhf and 1.5 hours on ppc64el, the slowest Debian architecture is riscv64 and the last successful build there took only 17.5h, so definitely not days. Your average modern developer-class laptop is going to take a lot less than riscv64 too.
Post reply on HN