So it’s just a script kiddie but running at 100X speed. Amazing "innovation"
Tailscale didn't stop the Hugging Face intrusion
161–170 of 239 posts
Re: Tailscale didn't stop the Hugging Face intrusion
#162This entire article reads like "hey we're tailscale and we also exist and we were tangentially involved in this".
Re: Tailscale didn't stop the Hugging Face intrusion
#163> No Tailscale vulnerability was found or exploited—we should have been able to prevent it anyway.
Which is it? A vulnerability doesn't have to be something not working as intended, it can be a lax security decision. Not making the safer path the easy path can qualify, if that's the standard you hold.
Here's hoping the era of humblebrag marketing by AI companies is just a phase.
Re: Tailscale didn't stop the Hugging Face intrusion
#164sorry I know off topic, is there any alternative VPN to tailscale. actually I have open Web socket to my mobile from my PC to stream screen record using ffmpeg. it's my own app . so anyway tailscale sometimes is slow . is there any alternative free service like tailscale vpn ? ( don't want to use anything like parsec etc , I want to use my own app)
CloudConnexa is the alternative from OpenVPN. 5 free seats. Built as a more traditional VPN, more bells and whistles, etc.
Re: Tailscale didn't stop the Hugging Face intrusion
#165Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
Next time we will - provided you further lock yourself into our ecosystem.
It's a bit harsh.
More to the point, and no longer specific to Tailscale: while plenty of companies do have rough "proof sketches" demonstrating the security of a setup under certain assumptions (hardness assumptions of the cryptosystem, threat model assumptions, ...) nobody is doing it formally to check with a general statement verifier. A lot of the advice and recommendations offered on the featured article may or may not be useful strategies for which properties are provable, and all the "security best practices" could be viewed as lego block lemmas from which system architects design a more reliable whole.
This means the community is doing redundant work. If people actually described and refined reusable assumptions say for the metamath verifier, people could download standard machine readable formalized threat models, share proofs that demonstrate the security (conditional on this or that assumption), or proofs that demonstrate the insecurity for specific ways an assumption might be broken, etc.
Whenever an incident happens we can visualize exactly the set of assumptions among which at least 1 must have been violated, so different groups suspect this or that assumption to be violated and thus collectively define variations of security proofs with different assumptions. This diversity allows us to zoom in on the faulty assumption over time as incidents are accumulated.
Re: Tailscale didn't stop the Hugging Face intrusion
#166One thing I hope becomes more common after this incident is companies publishing what they would change even when the root cause wasn't their software. Anyways, I appreciate articles like this because it's easy for companies these days to say "we weren't the vulnerable component" and just move on
Re: Tailscale didn't stop the Hugging Face intrusion
#167Earlier quoted context omitted.
This all has the -aire of theatre. OH NOES THE POWERFUL AI GOT OUT Then everyone coming out with humbled determination about working together to responsibly use and contain this powerful technology for the greater good (and profit margin). I will not believe marketing gimmickry is not a large part of what's going on with every one of these "incidents".
I think part of it is we need to stop looking at the 'oh no it got out' and the 'what did it do with the prompt when no one is looking'. At the end of the day the probability that an AI gets out is unity, what it does while out is far more important. The fact they are hacking into systems at superhuman levels, or writing cryptominers on their own hacked internal systems is a much more interesting and telling story of…
[dead]
Re: Tailscale didn't stop the Hugging Face intrusion
#168Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
And yet everyone seems to do it anyway. Fine for medium security, but maybe the product needs a high security mode that enforces inconvenient decisions?
I don’t think everyone does this at all. Maybe very small scale companies, but I would have expected a company like Huggingface to have better security standards.
As soon as you’re big enough to have dedicated devops people, you shouldn’t be doing this anymore.
Re: Tailscale didn't stop the Hugging Face intrusion
#169I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
This is spectacular marketing.
Re: Tailscale didn't stop the Hugging Face intrusion
#170> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would…
When everyone push on friday, and you have 400 CICD pipeline triggers spawning that many nodes. How do you know if this is unexpected ? Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard
The one thing which might have helped is the “external sandbox” detail: registering CI instances is noisy but it should be highly consistent so you could potentially alert on use outside of the expected environment (e.g. I have AWS STS sessions bound to a source IP or VPC, which really came in handy when Trivy was compromised).
I wonder whether it’d be possible to do something like export the list of legit K8s instances periodically so you’d be able to look for usage outside of those, but at that point something like that proxy approach would be less work and workload identity federation would be even easier.