Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Processing 40TB of code from 10M projects with a dedicated server and Go

boyter.org

Post reply on HN
Print view View original

31–40 of 83 posts

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#31

Post by WilliamEdward »

breakerbox wrote: post #29
GrumpyNl wrote: post #21

Earlier quoted context omitted.

I would like to know to, any insights?

The only thing I’ve put together from light searching on GitHub might be to do with the fact that 15 and s15 can be used to describe school years (s for spring 15 semester) and a lot of people post assignments to GitHub. 15 is also a common number in coding problem sets that people post to GitHub. It’s a stretch but it might be something to do with 2015 being a year that has a lot of coursework commits, and 15 is a p…

advent of code also came out in 2015... so, maybe?

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#32

Post by reacharavindh »

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging.

Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to adopt. Python misses out on a lot of use because of the inability to do this. And No! I do not want to pip install a lot of stuff on the servers just to be able to run this script once. Heck, some of these servers don't even have access to public internet to be able to pip install whatever.

Yes, I have looked into pyinstaller and Nuitka. They threw up some errors that were indicative of deeper issues that I didnt feel like a good use of my time to debug. I'd rather choose a language that has this as a priority/design goal instead.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#33

Post by jmngomes »

tylerl wrote: post #12

Can't you do all this same research in just a few seconds for like pennies, just by sending a few sql queries to Google's BigQuery? I'm pretty sure we've had stories about that here. https://medium.com/google-cloud/github-on-bigquery-analyze-a...

I partially agree with you; unless the goal was to "do this in Go", the choice of tools seems odd/inneficient.

Spark would have a been a simple option to do this kind of processing, with less lines of code, and could also run on "spare compute". Same goes for the "How does one process 10 million JSON files taking up just over 1 TB of disk space in an S3 bucket?": there are appropriate file formats for storing and querying big datasets, text/json is simply the least efficient option and likely the cause of the "$2.50 USD per query" number...

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#34

Post by [deleted] »

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

[deleted]

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#35

Post by EdwardDiego »

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

Yeah, I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter.

My initial thought was Python, but it needed a couple of 3rd party dependencies, and there wasn't an overly clean _and_ simple way to copy the script in from the mixin and run locally.

So I shrugged, rewrote my script in Go, and then used a multi-stage build to copy in the binary and nothing else.

Ending up with a single statically linked binary was cool, even if Go does some stuff that made my eyebrows quirk a tad (I still can't believe that an idiomatic set in Go is map[T]struct{}...)

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#36

Post by EdwardDiego »

brazzledazzle wrote: post #8
ojkelly wrote: post #4

As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.

People actually bike shed over .yaml vs .yml? If you thought tabs vs spaces was a useless debate...

All I'm saying mate, is that we don't pronounce it Yiml now, do we. :D

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#37

Post by XJ6 »

hendi_ wrote: post #30
wazoox wrote: post #26

Earlier quoted context omitted.

MS-DOS. The reason is MS-DOS.

Thank you for the EXPLAN~1!

If anyone's curious this references the 8.3 filename shortening that windows was even still doing recently.

Amusingly, that tiny bit of backward compatibility can lead to vulnerabilities as outlined here: https://www.acunetix.com/blog/articles/windows-short-8-3-fil...

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#38

Post by reubenmorais »

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#39

Post by eeZah7Ux »

reubenmorais wrote: post #38
reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

This is becoming a parody of software engineering.

People are now irrationally scared of using dynamic libraries, OS packages and even directories, like virtualenvs.

Instead, a simple solution is replaced with containers, or by rewriting tons of code in the new hyped language.

Are we trying to create job security through unnecessary complexity?

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#40

Post by bauruine »

robbya wrote: post #2

> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify. * https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/servic…

Or you could just put it on some Hetzner box. I would do that but I'm not sure how to contact him.
Post reply on HN