Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Getting 10TB of GitHub logs and extracting details of all users and repositories

trickest.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#21

Post by [deleted] »

jxramos wrote: post #9
ziml77 wrote: post #8

Earlier quoted context omitted.

I have a GitHub account under my real name, but recently I've started using GitHub under a couple of other names instead. There's so much stuff you do in public on GitHub that I want to avoid people doing exactly this kind of analysis on. I wish using multiple identities was at least some level of foolproof though. I have to be careful to configure my local copies of repos to use the correct username, masked email, a…

Private repos are still off limits for this sort of analysis correct?

[deleted]

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#22

Post by kfrzcode »

didntcheck wrote: post #3

I'm a big fan of neat statistical analyses, but when I look at the archive site https://www.gharchive.org/ , the overwhelming feeling I have is "creeped out". Taking periodic snapshots of repositories and their issues and wikis sounds good, but do we really need a log of every time someone watches an issue, and every commit message being irrevocably set on public record? That level of details on individual activity s…

Lol let me introduce you to a little organization called the National Security Agency, with their "creepy" periodic snapshots of much more intriguing datasets.

"Stellar Wind" is a good place to start.

Including, but of course not limited to, every communication made by any person within the United States (or outgoing) for the better part of two decades. Internet traffic, communications, all of it.

https://oig.justice.gov/reports/2015/PSP-09-18-15-vol-III.pd...

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#23

Post by kiririn »

lesquivemeau wrote: post #19
didntcheck wrote: post #3

I'm a big fan of neat statistical analyses, but when I look at the archive site https://www.gharchive.org/ , the overwhelming feeling I have is "creeped out". Taking periodic snapshots of repositories and their issues and wikis sounds good, but do we really need a log of every time someone watches an issue, and every commit message being irrevocably set on public record? That level of details on individual activity s…

Is this even GDPR compliant ?

GitHub is already a GDPR joke. They won’t delete anything less than an entire account

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#24

Post by awesome_dude »

kfrzcode wrote: post #20
zhfliz wrote: post #17

Earlier quoted context omitted.

note that this is technically against their TOS if not using paid accounts: > One person or legal entity may maintain no more than one free Account (if you choose to control a machine account as well, that's fine, but it can only be used for running a machine) https://docs.github.com/en/site-policy/github-terms/github-t...

internet would be better with total anonymity.

[flagged]

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#26

Post by giancarlostoro »

sleepytimetea wrote: post #25

The background with static noise really bothers me. Will have to skip reading till they provide a disable button.

On Firefox and even Microsoft Edge there is a "reader mode" option for most websites. I click on that often enough when I expect an article, to remove noise from ads.

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#27

Post by mplewis9z »

0x0 wrote: post #16
ziml77 wrote: post #8

Earlier quoted context omitted.

I have a GitHub account under my real name, but recently I've started using GitHub under a couple of other names instead. There's so much stuff you do in public on GitHub that I want to avoid people doing exactly this kind of analysis on. I wish using multiple identities was at least some level of foolproof though. I have to be careful to configure my local copies of repos to use the correct username, masked email, a…

You can have includeIf sections in your .gitconfig that applies only to things within a certain directory. So you if you create top level directories in your $HOME for your various identities, all you need to do is to make sure you are cloning into and working within the appropriate directory hierarchy.

I do this to separate personal and company repositories on the same machine, and it works flawlessly. An example config looks like:

~/.gitconfig:

```

[includeIf "gitdir:~/Personal/"]

        path = ~/personal.gitconfig
```

~/personal.gitconfig:

```

[user]

        email = {personal email address}
```

And you can have arbitrary numbers of "profiles" like this, as long as each is in their own directory.

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#28

Post by George83728 »

amusingimpala75 wrote: post #4
atum47 wrote: post #2

[flagged]

Even https://archive.ph/atw1q didn’t work correctly on the page, it just ceases to scroll after a point.

The website works fine unless you enable javascript. That's usually the way it is with these sort of things. The webdev or CMS creates a perfectly functional website using HTML and CSS, then some javascript is added to shit the whole thing up. Disable javascript by default for a better web experience.

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#29

Post by Firmwarrior »

awesome_dude wrote: post #24
kfrzcode wrote: post #20

Earlier quoted context omitted.

internet would be better with total anonymity.

[flagged]

Well, to be fair, some of the most vile hateful stuff I've ever seen in my life was posted on Facebook-powered comment sections under news articles, next to real names and pictures of smiling grandparents holding their grandchildren

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#30

Post by pcdevils »

lesquivemeau wrote: post #19
didntcheck wrote: post #3

I'm a big fan of neat statistical analyses, but when I look at the archive site https://www.gharchive.org/ , the overwhelming feeling I have is "creeped out". Taking periodic snapshots of repositories and their issues and wikis sounds good, but do we really need a log of every time someone watches an issue, and every commit message being irrevocably set on public record? That level of details on individual activity s…

Is this even GDPR compliant ?

Based on Serbia but operating the servers in Frankfurt. With the cavalier attitude to scraping and linking people's identifiable data without any sort of opt-in I had assumed it would be a US company.
Post reply on HN