Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Repositories held for ransom by using valid credentials

about.gitlab.com

Post reply on HN
Print view View original

131–140 of 158 posts

Re: Repositories held for ransom by using valid credentials

#131

Post by manigandham »

paulddraper wrote: post #95
soulofmischief wrote: post #53

Earlier quoted context omitted.

Security isn't rendered in absolutes. We have to assume some sheepish new employee somewhere is scared of approaching management about a mistake they made committing a secret, so they reverse the commit and pretend nothing ever happened. We have to try and mitigate damage from lapses in communication and protocol like that.

Yes and this is why Github handicapped their search so much. But at the end of the day, any secret you post publicly is compromised.

That's a very poor reason to not have good searching functionality considering the uses far outweighs the potential risk. I highly doubt it's the case.

Re: Repositories held for ransom by using valid credentials

#132

Post by manigandham »

1f60c wrote: post #33
avar wrote: post #17

> We believe that no data has been lost, unless the [...] GitLab copy was the only one. One difference between how GitLab and GitHub run their infrastructure is that GitLab doesn't keep reflogs, and uses git's default "gc" settings. As a result they won't have the data in question anymore in many cases[1]. Well, I don't 100% know that for sure, but it's the default configuration of their software, and I'm assuming th…

This tendency of Hacker News users to want to monetize everything is sickening.

Because nothing is free. You don't have to pay for it if it has no value, or you can do it yourself.

Re: Repositories held for ransom by using valid credentials

#133

Post by dang »

1f60c wrote: post #33
avar wrote: post #17

> We believe that no data has been lost, unless the [...] GitLab copy was the only one. One difference between how GitLab and GitHub run their infrastructure is that GitLab doesn't keep reflogs, and uses git's default "gc" settings. As a result they won't have the data in question anymore in many cases[1]. Well, I don't 100% know that for sure, but it's the default configuration of their software, and I'm assuming th…

This tendency of Hacker News users to want to monetize everything is sickening.

Maybe so, but please don't post unsubstantive comments or get flamebaity here.

https://news.ycombinator.com/newsguidelines.html

(Nearly all such generalizations about HN users are just sample bias anyhow.)

Re: Repositories held for ransom by using valid credentials

#134

Post by kelnage »

weinzierl wrote: post #124
bostik wrote: post #102

Earlier quoted context omitted.

And a hundred times so for any public repos. There are bots feeding on the GitHub firehose, scavenging for accidentally committed credentials. A few years back (2015 or so) the average time from push-to-repo to AWS account compromise was 6 minutes . Surely that time has only gone down, and the number of different credentials identified has gone up.

> A few years back (2015 or so) the average time from push-to-repo to AWS account compromise was 6 minutes. Surely that time has only gone down, and the number of different credentials identified has gone up. I don't doubt that a second and I'd like to use that as a quote. I'd like to be prepared if someone doubts it, so: Do you have a primary source for this?

This paper may be relevant to your interests: https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-chara...

Re: Repositories held for ransom by using valid credentials

#135

Post by zifnab06 »

gitlab-security wrote: post #111
mikeash wrote: post #93

Earlier quoted context omitted.

That is a crazy-ass quote. “We believe that no data has been lost... well, except for the data we keep. But you weren’t actually relying on us to save any data, right?” I know, back up everything at least twice. But still, when somebody loses one of your copies, they don’t get to say “it’s cool, no data was lost, you have other copies, right?”

To provide additional context, on GitLab.com, we maintain two weeks' backups. The last time we restored a single project repo, it was a significant effort that utilized many hours of an SRE's time to complete.

Does support have the ability to restore keeparound refs? Internally (at least as of late 10 series), anything show in the UI (ie, merge requests) is also copied to refs/keeparound/sha1, which isn't presented to users.

Re: Repositories held for ransom by using valid credentials

#136

Post by scraptor »

weinzierl wrote: post #124
bostik wrote: post #102

Earlier quoted context omitted.

And a hundred times so for any public repos. There are bots feeding on the GitHub firehose, scavenging for accidentally committed credentials. A few years back (2015 or so) the average time from push-to-repo to AWS account compromise was 6 minutes . Surely that time has only gone down, and the number of different credentials identified has gone up.

> A few years back (2015 or so) the average time from push-to-repo to AWS account compromise was 6 minutes. Surely that time has only gone down, and the number of different credentials identified has gone up. I don't doubt that a second and I'd like to use that as a quote. I'd like to be prepared if someone doubts it, so: Do you have a primary source for this?

Blog post summary: https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-chara...

Source if blog is unavailable: https://www.ndss-symposium.org/wp-content/uploads/2019/02/nd...

Re: Repositories held for ransom by using valid credentials

#137

Post by scarface74 »

tjr225 wrote: post #119
username223 wrote: post #110

Earlier quoted context omitted.

> the average time from push-to-repo to AWS account compromise was 6 minutes . Wow, I didn't realize it had become so efficient, but I shouldn't be surprised. I never really understood the value in hosting non-public software in the public, and if it's open source, it shouldn't be getting anywhere near secrets that can be used to extract money from its developers. I remember thinking, back when it became trendy for p…

I had accidentally pushed an AWS credential out a month or two ago- within about a minute and a half AWS had disabled the IAM user, and automatically emailed me(as well as my entire org- how embarrassing!)- when we were going through the access logs it looked like it had taken only a minute and a half longer for some other, presumably malicious, system to attempt to access my compromised user. Probably between 2 or 3…

Why have credentials anywhere outside of the .aws directory in your home directory? When developing locally all of the SDKs will read them from there and when deploying to AWS, the SDK will get them from the attached role.

Re: Repositories held for ransom by using valid credentials

#138

Post by gitlab-security »

snowwolf wrote: post #122

While the fault lies with the users for not following security best practices, including enabling 2FA there are things gitlab/any site can do to help defend against these sorts of attacks. Some suggestions: Treat logins from datacenters as suspicious. (In this case the IP block identified belongs to World Hosting Farm Limited). Treat logins from a new/different ISP as suspicious. Limit access to the account and verif…

Thank you for your feedback and suggestions. Unfortunately, for each of these proposals, we're likely to have users asking us why we are restricting and/or blocking access.

A better defense-in-depth strategy would be to scan each public repo for credentials, and act accordingly when credentials are discovered in repos. We are working on this strategy, currently.

Re: Repositories held for ransom by using valid credentials

#139

Post by snowwolf »

gitlab-security wrote: post #138
snowwolf wrote: post #122

While the fault lies with the users for not following security best practices, including enabling 2FA there are things gitlab/any site can do to help defend against these sorts of attacks. Some suggestions: Treat logins from datacenters as suspicious. (In this case the IP block identified belongs to World Hosting Farm Limited). Treat logins from a new/different ISP as suspicious. Limit access to the account and verif…

Thank you for your feedback and suggestions. Unfortunately, for each of these proposals, we're likely to have users asking us why we are restricting and/or blocking access. A better defense-in-depth strategy would be to scan each public repo for credentials, and act accordingly when credentials are discovered in repos. We are working on this strategy, currently.

That doesn’t help stop the attacks using breach lists that are even more prevalent.

You could start with email warnings of suspicious activity and fine tune the model parameters based on feedback from false positives. But generally a login from a device that has no previous cookie, from an ASN the account has never used before, especially if that ASN is a known data center, that then immediately attempts a destructive action, should be a pretty big warning flag.

Re: Repositories held for ransom by using valid credentials

#140

Post by tjr225 »

scarface74 wrote: post #137
tjr225 wrote: post #119

Earlier quoted context omitted.

I had accidentally pushed an AWS credential out a month or two ago- within about a minute and a half AWS had disabled the IAM user, and automatically emailed me(as well as my entire org- how embarrassing!)- when we were going through the access logs it looked like it had taken only a minute and a half longer for some other, presumably malicious, system to attempt to access my compromised user. Probably between 2 or 3…

Why have credentials anywhere outside of the .aws directory in your home directory? When developing locally all of the SDKs will read them from there and when deploying to AWS, the SDK will get them from the attached role.

I understand what the best practices are, it was a total mistake- I never even intended to push what I did to github.
Post reply on HN