Processing 40TB of code from 10M projects with a dedicated server and Go
1–10 of 83 posts
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#2> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size.
Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify.
* https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/services/open-datasets/
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#3Missing content for:
https://boyter.org/posts/an-informal-survey-of-10-million-gi...
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#4As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub.
That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private.
Also, it look’s like 20% of code is comments. Which feels just about right.
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#5As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.
Surely copy & paste played a major role in this.
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#6> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify. * https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/servic…
I similarly have a 127GB tar.xz of about 2TB of code/files from the top GitHub repositories (filtered by highest starred repositories per primary language) as of around August 2017 if anyone wants it or wants to host it.
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#7As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.
Surely copy & paste played a major role in this.
Yep that and forking. But if you fork a repo and change a small thing, I’d it still a duplicate of the other repo?
I don’t think the raw number is what’s so impressive, as much as the fact that there’s more code public on GitHub than I could comprehend in a lifetime.
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#8As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.
People actually bike shed over .yaml vs .yml? If you thought tabs vs spaces was a useless debate...
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#9This is great. Btw. seeing "jquery" most-common filename 20,015,171 times - it probably skews js LoC (artificial LoC of dependencies).
Re: Processing 40TB of code from 10M projects with a dedicated server and Go
#10This is a cool project and a great write-up. Some of the complexity numbers struck me as pretty off though. I know he caveats you can only compare files of the same language, but I took a quick look at the code[1] and at least as of now it looks like the complexity statements are just kind of copy/pasted, so keywords not seen in Java/C such as "match" are missing from Rust, OCaml, and Scala, "case" is missing from Haskell and Elixir etc, causing these languages to be much lower in complexity than they should be just based on standard control flow statements.
[1] https://github.com/boyter/scc/blob/master/languages.json