Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
1–10 of 16 posts
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#2Oops linked to an anchor in the body and cannot edit. Correct link: https://boyter.org/posts/an-informal-survey-of-10-million-gi...
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#3[deleted]
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#4Previously:
https://news.ycombinator.com/item?id=21121735 (80 comments)
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#5[deleted]
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#6Does anyone know why "pawn" is considered an offensive word? It seems likely it ranks highly on the offensive word list because of chess engines.
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#7Does anyone know why "pawn" is considered an offensive word? It seems likely it ranks highly on the offensive word list because of chess engines.
Phonetically it sounds like "porn".
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#8Does anyone know why "pawn" is considered an offensive word? It seems likely it ranks highly on the offensive word list because of chess engines.
I also noticed "knob" and "xxx" near the top of that list (and others) for which I understand why they are considered "offensive" in some contexts, but in code those are quite non-offensive (unless, of course, the context is offensive, haha).
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#9>>> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size.
Anyone knows if it is possible to download similar data set for youtube and reddit? I have ideas for search engine based on it, but I don't want to write/maintain scraper scripts.
Re: Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)
#10>>> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Anyone knows if it is possible to download similar data set for youtube and reddit? I have ideas for search engine based on it, but I don't want to write/maintain scraper scripts.
I have most of the historical reddit data except ~year I use to train ML models. Let me see if I can find a public link for you...