Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Processing 40TB of code from 10M projects with a dedicated server and Go

boyter.org

Post reply on HN
Print view View original

61–70 of 83 posts

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#61

Post by penagwin »

tseabra wrote: post #49
mtberatwork wrote: post #47

Earlier quoted context omitted.

> The server might have the wrong Python version available That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.

You'd need to install the right Python version though, and even with something like pyenv you might need to install system packages. It really isn't as portable as one might wish.

This is my exact situation right now. Our website is made in Django, and we use Pyenv to manage our python versions. We'll likely be going to route of a docker image/ pyenv install script. It takes trial/error to figure out some of the packages you need if they aren't installed already, a single binary would be far easier for sure.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#62

Post by john-radio »

Groxx wrote: post #11

First I've heard of "Taco Bell Programming"[1]. I love it, this needs to become a normal part of my lexicon. Also heck yes, it completely mirrors my experience - a few dozen tools combine in fairly simple ways to make absurdly useful results. The problem is finding the useful N-dozen elemental tools you'll use. [1]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python.

And with one dumb word choice, I suddenly can't share this otherwise good piece with most audiences.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#63

Post by tomatocracy »

wazoox wrote: post #26
pixelbash wrote: post #17

Earlier quoted context omitted.

Never been comfortable with extension names longer than 3 letters long for some reason..

MS-DOS. The reason is MS-DOS.

It goes back further than that - DOS got the idea of 8.3 filenames from CP/M and before that DEC used three letter file extensions on its minicomputer operating systems. The idea arguably goes back even further into the mainframes of the 1960s although it gets a bit messier then to trace it.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#64

Post by steeleduncan »

twic wrote: post #55
skocznymroczny wrote: post #51

Earlier quoted context omitted.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

I am an utter noob at Python packaging, but i would like to learn more, so excuse the basic question, but: why doesn't it work? And is there anything in the ecosystem which is like that, but does work? I had heard that the Python packaging and deployment story had got a lot better in recent years, but this sounds like it still falls far short of table stakes.

Its not that Python packaging doesn't work, or that it is impossible to automate, but whatever you do it requires nontrivial time and effort to get right.

The point is that the static binary created by go is a single file that you copy into place and run. It is a lot simpler.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#65

Post by catacombs »

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

> The single binary deployment makes it a lot easier for users to adopt.

I've had some success making a Python file an executable. Though, I do understand your gripe with needing to install dependencies.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#66

Post by homonculus1 »

john-radio wrote: post #62
Groxx wrote: post #11

First I've heard of "Taco Bell Programming"[1]. I love it, this needs to become a normal part of my lexicon. Also heck yes, it completely mirrors my experience - a few dozen tools combine in fairly simple ways to make absurdly useful results. The problem is finding the useful N-dozen elemental tools you'll use. [1]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. And with one dumb word choice, I suddenly can't share this otherwise good piece with most audiences.

Seriously? That's no harsher language than calling someone a dick. Do you have to keep an ultra-clinical PR image for some reason or are we really that puritanical?

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#67

Post by turtlebits »

skocznymroczny wrote: post #51
reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

You just need a venv with the right architecture, which is fairly trivial. You can then copy the venv directly over.

See this repo: https://github.com/unixtreme/D3Edit. It has Linux/Mac/Windows virtualenvs and it works without any additional setup

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#68

Post by AdamJacobMuller »

skocznymroczny wrote: post #51
reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

I've done things in a way that it doesn't.

Basically build a virtualenv and then pack that into a .deb, install the .deb on the target machine and you have a self-contained package that requires no external resources to install.

That said, using golang is WAAAAY simpler. The virtualenv/deb solution, while it works, is very sketch.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#70

Post by dmit »

breakerbox wrote: post #20

I thought the most common file names had some peculiar results. > the filename “15” & “s15” made the top 50 most popular filenames Anyone know why? https://boyter.org/posts/an-informal-survey-of-10-million-gi...

s15 seems to be skewed by a bunch of JavaScript projects including Font Awesome, which stores the bathtub icon as s15.js for some reason (note that s1 through s14 are unused, so it's not just consecutive numeric ids). https://fontawesome.com/v4.7.0/icon/bath

Plain 15 is trickier. GitHub's search is too fuzzy to see a pattern at a quick glance, but again it seems that JavaScript is the culprit with a disproportionately high number of filename:15 results compared to filename:14 or filename:16.

Post reply on HN