Live data from Hacker News

Getting 10TB of GitHub logs and extracting details of all users and repositories

trickest.com

41–50 of 61 posts

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#41
post #34

Earlier quoted context omitted.

internet would be better with total anonymity.

That's one of my favorite parts of reddit. Accounts are just pseudonyms, and you can generate as many as you want. I personally generate a new one every few months, which helps keep too much identifying data from building up over time.

and they're all identifiably attached to your metadata package(s)

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#42
The article leaves a bitter taste of unnecessary complexity. Data engineering should not be hard.

For example, you can load the GitHub Archive to ClickHouse, and it will be accessible with interactive real-time queries: https://ghe.clickhouse.tech/

See also https://til.simonwillison.net/clickhouse/github-explorer

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#43

Was this written by GPT? I was quite interested in the topic of the article but I started to get the brain fog I associate with parsing ChatGPTs convoluted sentences.

It also seems like a thinly veiled piece of product marketing.

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#44
post #34

Earlier quoted context omitted.

That's one of my favorite parts of reddit. Accounts are just pseudonyms, and you can generate as many as you want. I personally generate a new one every few months, which helps keep too much identifying data from building up over time.

and they're all identifiably attached to your metadata package(s)

Which, at the very least, isn't public to everyone. So, not perfect, but better than single public account.

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#45
post #13

Earlier quoted context omitted.

[flagged]

[flagged]

Yes but the specific ways that large companies like Microsoft violate user privacy tends not to include posting sensitive user data on the open internet. Especially not for business customers.

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#46

Was this written by GPT? I was quite interested in the topic of the article but I started to get the brain fog I associate with parsing ChatGPTs convoluted sentences.

Agreed... I wanted to understand what it was all about, but really struggled to follow. They talk about the whole thing taking around 24 hours, but some part took over 30. Also that it ran on a 4GB of RAM machine, but they needed larger ones to do all the parsing.

Also in the end, unsure of what the actual results are. Maybe I missed clicking on something.

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#48
post #17

Earlier quoted context omitted.

note that this is technically against their TOS if not using paid accounts: > One person or legal entity may maintain no more than one free Account (if you choose to control a machine account as well, that's fine, but it can only be used for running a machine) https://docs.github.com/en/site-policy/github-terms/github-t...

internet would be better with total anonymity.

it'll be nice to have options

nostr is allowing for more options

and Jack Dorsey happens to have put up a 1 billion sats bounty to """Create a “complete” Nostr-based suite of git tools, such that projects like bitcoin-core are sufficiently confident to move away from GitHub."""

https://bountsr.org/nostr-based-github/

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#49

Was this written by GPT? I was quite interested in the topic of the article but I started to get the brain fog I associate with parsing ChatGPTs convoluted sentences.

It also seems like a thinly veiled piece of product marketing.

This

Re: Getting 10TB of GitHub logs and extracting details of all users and repositories

#50

I'm a big fan of neat statistical analyses, but when I look at the archive site https://www.gharchive.org/ , the overwhelming feeling I have is "creeped out". Taking periodic snapshots of repositories and their issues and wikis sounds good, but do we really need a log of every time someone watches an issue, and every commit message being irrevocably set on public record? That level of details on individual activity s…

Don't blame the archiver who is only doing what is allowed of them by Github. If you don't want to be public don't be public. Other archivers aren't making themselves publicly known but have the data all the same.

Just because you can perve on people at a nudest beach doesn't mean you should.

Society frequently has norms and unwritten rules that we usually follow. Not every rule needs to be written down as law.

Post reply on HN