Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rubygems.org AWS Root Access Event – September 2025

rubycentral.org

Post reply on HN
Print view View original

51–60 of 179 posts

Re: Rubygems.org AWS Root Access Event – September 2025

#51

Post by LamaOfRuin »

yargzeblog wrote: post #41
ctoth wrote: post #38

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing. They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window. You ca…

IMO the only way to avoid doing a total rebuild is to have Andre Arko: 1. Admit that he was the unauthorized actor (which means he's probably admitting to a crime?) 2. Have him attest he didn't exfil or modify the integrity of service while committing a crime. If I was Ruby Central I would give clemency on #1 in exchange for #2 and I think #2 helps Andre Arko.

If Andre doing that was criminal, it seems quite possible that their original takeover of the github organization was also criminal?

I have been waiting to hear if there would be any civil action on it since it's not at all clear they had any rights to do most of what they did.

Re: Rubygems.org AWS Root Access Event – September 2025

#52

Post by shevy-java »

I'd recommend to people to wait for a response - RubyCentral spins up a gazillion accusations right now and has been in the last days (and, it is also incomplete, because why did they fire every dev here and placed Marty Haught in charge specifically? They never were able to logically explain this; plus, why didn't they release this write-up before? It feels very strange to wait here; they could have clarified things before, but to me it seems they kind of waited and then tried to come up with some explanation that, to me, makes no real sense).

I also highly recommend to not accept RubyCentral's current strategy to post very isolated emails and insinuate that "this is the ultimate, final proof". We all know that email conversation often requires lots of emails. So doing a piecemail release really feels strange. Plus, there also were in-person meetings - why does RubyCentral not release what was discussed here? Was there a conflict of interest due to financial pressure?

Also, as was already pointed out, RubyCentral went lawyering up already - see discussions on reddit. Is this really the transparency we as users and developers want to see? This is blowing up by the day and no matter from which side you want to look at it, RubyCentral sits at the center; or, at the very least, made numerous mistakes, tries to cover past mistakes by ... making more mistakes. I think it would be better to dissolve RubyCentral. Let's start from a clean state here; let's find rules of engagement that doesn't put rich corporations atop the whole ecosystem.

Last but not least - this tactical slandering is really annoying. If they have factual evidence, they need to bring the matter to a court; if they don't, they need to stop slandering people. To my knowledge RubyCentral hasn't yet started a court case, and I have a slight suspicious that they also will not, because we, as the general public, would then demand COMPLETE transparency, including ALL of RubyCentral's members and their activities here. So my recommendation is: wait for a while, let those accused respond.

Re: Rubygems.org AWS Root Access Event – September 2025

#53

Post by oneplane »

akerl_ wrote: post #47
zrail wrote: post #32

Earlier quoted context omitted.

Resetting the root password requires proving access to the email address associated with the root account. It also leaves a massive papertrail.

By "massive papertrail" do you mean "a pair of emails to the associated address"?

No, a massive amount of CloudTrail logs.

Re: Rubygems.org AWS Root Access Event – September 2025

#54

Post by oneplane »

nerdjon wrote: post #26
oneplane wrote: post #23

Earlier quoted context omitted.

You don't need the root account, unless you need to bypass all policies. In such a scenario, you a use the root access reset flow instead, reducing standing access. As for other flows (break glass, non-SSO etc), that can all be handled using IAM users. You'd normally use SAML to assume a role, but when SSO is down you'd use your fallback IAM user and then assume the role you need. As for how you disable the root acco…

To me that sounds like security by obscurity not actual security. If you have the ability to go through the reset flow than then why is that much different than the username and password being available to a limited sets of users. That would not have prevented this from happening if the determination was made that all 3 of these users need the ability to possibly get into root. As far as having an IAM user, I fail to…

Not using root means not bypassing policies. There is no way to not bypass all policies. So yes, never using root makes that issue go away completely.

As for all the other stuff: what it does is it creates distinct identities with distinct credentials and distinct policies. It means that there is no multi-party rotation requires, you can nuke the identity and credentials of a specific person and be done with it. So again, a real solution to a real problem.

Re: Rubygems.org AWS Root Access Event – September 2025

#55

Post by shevy-java »

ctoth wrote: post #38

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing. They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window. You ca…

You are actually right, I didn't think of this before.

How can they ensure that nobody else did any tampering?

It seems RubyCentral did not think this through completely.

Re: Rubygems.org AWS Root Access Event – September 2025

#56

Post by TehCorwiz »

jnewland wrote: post #46

This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:" > 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or contr…

Right?! Did nobody there think to actually disable the accounts? These are the people who are harping about "security" being the reason for the ham-fisted takeover of the source repos, but they didn't secure the production infrastructure?

Re: Rubygems.org AWS Root Access Event – September 2025

#57

Post by busterarm »

shevy-java wrote: post #55
ctoth wrote: post #38

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing. They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window. You ca…

You are actually right, I didn't think of this before. How can they ensure that nobody else did any tampering? It seems RubyCentral did not think this through completely.

You can set up EC2 instances in a way that that just having AWS root access doesn't give you ssh/console access to the instances. You can still do things like Run Command but that leaves a very obvious trail (although even this is preventable with enough effort).

Also you can enable cloudtrail log validation which can ensure you know if you're looking at tampered logs or not.

Really it all depends on how their accounts are set up. Unless you know the operational details you can't make a call here.

I've run a multi-million dollar/year AWS Org for the last decade or so and setting things up this way is kind of brass tacks.

Re: Rubygems.org AWS Root Access Event – September 2025

#58

Post by dismalaf »

shevy-java wrote: post #52

I'd recommend to people to wait for a response - RubyCentral spins up a gazillion accusations right now and has been in the last days (and, it is also incomplete, because why did they fire every dev here and placed Marty Haught in charge specifically? They never were able to logically explain this; plus, why didn't they release this write-up before? It feels very strange to wait here; they could have clarified things…

> let those accused respond.

Literally all we've heard so far is from the other side...

> If they have factual evidence, they need to bring the matter to a court

I'd be surprised if they aren't. This post feels very much like the amount of disclosure a lawyer would recommend to reassure stakeholders.

> rules of engagement that doesn't put rich corporations atop the whole ecosystem

Right now the only thing stopping us all from being held hostage by rogue maintainers is a rich corporation.

Re: Rubygems.org AWS Root Access Event – September 2025

#59

Post by jhealy »

the_mitsuhiko wrote: post #24
flumpcakes wrote: post #21

Earlier quoted context omitted.

I'm now questioning my sanity but I thought you could disable login for the root account in AWS.

Since there are certain operations that can only be done with the root account, there is no way to disable access to it.

Since 2024 you can disable the root credentials on all accounts except the Organization management account: https://aws.amazon.com/blogs/aws/centrally-managing-root-acc...

I don't think the post mortem details whether the root access was on the org management account or an org member account.

Re: Rubygems.org AWS Root Access Event – September 2025

#60

Post by ctoth »

akerl_ wrote: post #50
ctoth wrote: post #38

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing. They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window. You ca…

Given the context of the post, it seems like "Enabled AWS CloudTrail, GuardDuty, and DataDog alerting" means "enabled alerts via CloudTrail, GuardDuty, and Datadog", not "enabled Cloudtrail logging". Otherwise the comment about reviewing Cloudtrail wouldn't make sense.

So the attacker turns logging off (was log file validation enabled? usually isn't in Terraform ) which does not fire an alert because there is no alerting. Then does their bad stuff ... Then modifies the logs (which are in an S3 bucket on the compromised account, remember!) Then they turn logging on? The whole point is alerts go outside AWS. They go to like, your inbox or pagerduty or whatever. If they had no alerts then what use are their logs, which could have been modified? Do you think they set up cross-account logging or had enable_log_file_validation set to true?
Post reply on HN