Live data from Hacker News

Processing 40TB of code from 10M projects with a dedicated server and Go

boyter.org

21–30 of 83 posts

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#21

I thought the most common file names had some peculiar results. > the filename “15” & “s15” made the top 50 most popular filenames Anyone know why? https://boyter.org/posts/an-informal-survey-of-10-million-gi...

I would like to know to, any insights?

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#22
post #2

> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify. * https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/servic…

Make a bit torrent

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#24
post #2

> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify. * https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/servic…

You could also use an S3 or GCS “requester pays” bucket, which means you never get charged for data transfer. Would cost about $2/month for the 83GB storage, but that’s it.

https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPay...

https://cloud.google.com/storage/docs/requester-pays

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#26

Earlier quoted context omitted.

People actually bike shed over .yaml vs .yml? If you thought tabs vs spaces was a useless debate...

Never been comfortable with extension names longer than 3 letters long for some reason..

MS-DOS. The reason is MS-DOS.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#28

This is a cool project and a great write-up. Some of the complexity numbers struck me as pretty off though. I know he caveats you can only compare files of the same language, but I took a quick look at the code[1] and at least as of now it looks like the complexity statements are just kind of copy/pasted, so keywords not seen in Java/C such as "match" are missing from Rust, OCaml, and Scala, "case" is missing from Ha…

In all seriousness if you are a language expert for those languages please submit a PR or at least raise an issue saying what keywords should be in there. I am happy to include it into the next release should it produce more accurate numbers.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#29

I thought the most common file names had some peculiar results. > the filename “15” & “s15” made the top 50 most popular filenames Anyone know why? https://boyter.org/posts/an-informal-survey-of-10-million-gi...

I would like to know to, any insights?

The only thing I’ve put together from light searching on GitHub might be to do with the fact that 15 and s15 can be used to describe school years (s for spring 15 semester) and a lot of people post assignments to GitHub.

15 is also a common number in coding problem sets that people post to GitHub. It’s a stretch but it might be something to do with 2015 being a year that has a lot of coursework commits, and 15 is a pretty low number (i.e. higher probability a student posts solutions to #1-15 than #1-19).

The same goes for schoolwork. Many people do online MIT and Stanford courses that are from previous years ~2012-2016, but there has been less time elapsed for students to post answers from 2016,17,18,19)

This is mostly conjecture, so I hope someone has a better answer!

Post reply on HN