Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Processing 40TB of code from 10M projects with a dedicated server and Go

boyter.org

Post reply on HN
Print view View original

11–20 of 83 posts

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#11

Post by Groxx »

First I've heard of "Taco Bell Programming"[1]. I love it, this needs to become a normal part of my lexicon. Also heck yes, it completely mirrors my experience - a few dozen tools combine in fairly simple ways to make absurdly useful results. The problem is finding the useful N-dozen elemental tools you'll use.

[1]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#13

Post by foobarqwertz »

taf2 wrote: post #5
ojkelly wrote: post #4

As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.

Surely copy & paste played a major role in this.

I am waiting for the results showing how much copy pasta is on the plate.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#14

Post by jtwaleson »

Haha, is this a coincidence or did you take inspiration from my 2016 blog post?

"Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS"

http://blog.waleson.com/2016/01/parsing-10tb-of-metadata-26m...

We used a lot of the same tech: python, golang, S3, 32 core machines. Anyway, nice read and good use of Hetzner.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#17

Post by pixelbash »

brazzledazzle wrote: post #8
ojkelly wrote: post #4

As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.

People actually bike shed over .yaml vs .yml? If you thought tabs vs spaces was a useless debate...

Never been comfortable with extension names longer than 3 letters long for some reason..

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#18

Post by fastball »

ojkelly wrote: post #4

As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.

.yml only makes sense in a word where every other extension is three letters.

With variably length extensions, there isn't a good reason to not just use the full name.

Post reply on HN