Live data from Hacker News

Incident Report: Inadvertent Private Repository Disclosure

github.com

21–30 of 43 posts

Re: Incident Report: Inadvertent Private Repository Disclosure

#21
post #6

Earlier quoted context omitted.

Don't you mean on-premise?

Datacentre's have outstanding track records, and if you secure your box correctly there are few ways to compromise it. On-premise will either be incredibly costly or missing key protections or infrastructure. Self-hosted git (through the many installable git servers or raw git) running on a correctly sized box is almost certainly the way to go

This github vulnerability has nothing to do with insecure box. It has to do with a bad application logic. This can happen anywhere - self-hosted or not.

Re: Incident Report: Inadvertent Private Repository Disclosure

#22
One of the most striking things about this report is the scale that GitHub has now reached: the whole incident apparently lasted only 10 minutes, but during that time 17 million requests were sent to their git proxy.

It's obviously unfortunate in this case, since even a relatively small and quickly fixed bug affecting a tiny proportion of requests still had serious consequences.

However, it's a remarkable achievement (if also a little terrifying for the software development industry from a single-point-of-failure perspective).

Re: Incident Report: Inadvertent Private Repository Disclosure

#23

Earlier quoted context omitted.

Datacentre's have outstanding track records, and if you secure your box correctly there are few ways to compromise it. On-premise will either be incredibly costly or missing key protections or infrastructure. Self-hosted git (through the many installable git servers or raw git) running on a correctly sized box is almost certainly the way to go

This github vulnerability has nothing to do with insecure box. It has to do with a bad application logic. This can happen anywhere - self-hosted or not.

That is true. But, if I'm running my own instance, the probability is that it doesn't matter if someone else gets access via bad application logic. Everybody is probably employed by the same company.

It's a difference of degree: compromising my self-hosted or on-premise server means that somebody already in my employ has more access than they should. If I'm a small organization, that probably doesn't matter. If I'm a big organization, I probably have an IT staff to deal with this and the people involved are still "nominally" under my control.

The github mistake means that people completely unrelated to my repository can get access.

Re: Incident Report: Inadvertent Private Repository Disclosure

#24
post #5

I approve of the handling, but this just underscores why you want self-hosted instances.

If you have private repositories that should never be public, no matter what, which isn't true for most users of private repositories, then:

Given that git provides many transports and ways to push commits around, I agree. If you have to be safe, there's no reason to use github or a self-hosted git collaboration service (gitlab, etc) on a publicly accessible server, regardless of access control measures. If you really need to have sources on a remote machine, you can limit the potential damage by only sharing archives of a certain revision, without history.

I know many will dismiss it, but if you're serious about the repositories being private, then the most you make them accessible is via an on-premises hosted gitlab instance, which is local to the company network, not accessible via the public network, and only allowed to, if you want, by dialing into VPN first. Then, to be safe, you null-route anything but the VPN traffic on the connected off-premises developer machine.

Access keys get stolen, just like SSH keys are, so you need to use a VPN service that requires additional security like the use of OTP key generators or similar measures.

This probably sounds like a hassle in the day and age of people just going for the comfort of private github or gitlab repositories, but it's what companies have been doing for almost 20 years as standard practice.

You cannot consider any git repository, even on your own root server, safe to keep private code on. CIOs would argue against that practice for good reason. The same CIOs require work laptops to encrypt all data.

If you don't need to be that serious, then an incident like this should be planned and accounted for as part of using such hosting, and shouldn't be a big deal.

Re: Incident Report: Inadvertent Private Repository Disclosure

#25

Next step: setup development system ?! Surely they do some end-to-end testing?

They state in the post that of 17 million requests to their git-proxy server, only 230 of those requests could be identified as successful responses to incorrect data/repos, at a percent of 0.0013%.

I don't know of anyone that would recommend creating tests, even integration tests, that hammers a service to check to see if something like one hundredths of one percent of requests returns invalid data. If anything, the fact that a script is hammering a service that probably (in a Dev or QA environment) has much less data in it's database and file stores, and much less protection (like load balancing and caching) than it would in production would generate more false positives than it would generate in substantial data disclosure regression defects.

Re: Incident Report: Inadvertent Private Repository Disclosure

#26
post #5

I approve of the handling, but this just underscores why you want self-hosted instances.

We've had a couple of VPS provider admin panel compromises, Shellshock and other showstopper remote vulnerabilities for privately administered servers that don't get constant professional attention, etc. It's possible, but not obvious.

Re: Incident Report: Inadvertent Private Repository Disclosure

#28

How did they become aware of the bug so quickly (<10 minutes)? Unless I'm missing something from the report, it doesn't say.

The bug trigger a flood of errors.

> The impact of this bug for most queries was a malformed response, which errored and caused a near immediate rollback.

Re: Incident Report: Inadvertent Private Repository Disclosure

#29

Earlier quoted context omitted.

I wrote the script in question and actually used a simple shannon entropy value. ( http://codereview.stackexchange.com/questions/868/calculatin... ). It worked well enough help rule out several problem spaces.

Would you mind posting the script? I'd love to run it against our codebase and see what it comes up with. It might be a fun thing to open source as part of a "I've inherited a project, what now?" toolkit that helps you decide what to fix.

IIRC Instagram released a plugin for Bandit (the OpenStack static analyzer for Python) that does this.
Post reply on HN