Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Processing 40TB of code from 10M projects with a dedicated server and Go

boyter.org

Post reply on HN
Print view View original

71–80 of 83 posts

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#71

Post by sswezey »

turtlebits wrote: post #67
skocznymroczny wrote: post #51

Earlier quoted context omitted.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

You just need a venv with the right architecture, which is fairly trivial. You can then copy the venv directly over. See this repo: https://github.com/unixtreme/D3Edit . It has Linux/Mac/Windows virtualenvs and it works without any additional setup

I think what people are arguing is deploying a single binary is even more trivial than a "venv with the right architecture". Deploying one file is going to be easier than deploying a set of files.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#72

Post by oblio »

mtberatwork wrote: post #47
theshrike79 wrote: post #46

Earlier quoted context omitted.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous. The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera. Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

> The server might have the wrong Python version available That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.

Portable means something else. Portable would mean that I could ship the virtualenv. Python missed the forest for the trees.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#73

Post by theshrike79 »

folkrav wrote: post #53
theshrike79 wrote: post #46

Earlier quoted context omitted.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous. The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera. Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

Why is it ridiculous? You'll need the related libraries either case, and by default virtualenv uses symlinks. The "full virtualenv" part makes it sound like it's heavy, or something like that.

If you do it in a language that provides static binaries or an equivalent, you don't need to pull dozens of libraries with possibly hundreds of files just to run a single script.

Imagine a situation where you'd need to perform an operation on a hundred servers, would you either transfer one static exacutable or build an virtualenv in every machine and download correct versions all of the related libraries?

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#74

Post by theshrike79 »

mtberatwork wrote: post #47
theshrike79 wrote: post #46

Earlier quoted context omitted.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous. The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera. Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

> The server might have the wrong Python version available That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.

I still need to have python installed on the machine and download all of the libraries needed.

Not all operating systems carry Python >3.3 by default, that would mean installing backports or unofficial repositories on production machines.

Or you can use a static binary that just works.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#75

Post by folkrav »

theshrike79 wrote: post #73
folkrav wrote: post #53

Earlier quoted context omitted.

Why is it ridiculous? You'll need the related libraries either case, and by default virtualenv uses symlinks. The "full virtualenv" part makes it sound like it's heavy, or something like that.

If you do it in a language that provides static binaries or an equivalent, you don't need to pull dozens of libraries with possibly hundreds of files just to run a single script. Imagine a situation where you'd need to perform an operation on a hundred servers, would you either transfer one static exacutable or build an virtualenv in every machine and download correct versions all of the related libraries?

If I controlled the environment, I'd probably build a Docker image and deploy that. Otherwise yes, This kind of extreme example isn't very representative of the initial situation though. And you're also talking about switching up languages, which isn't always possible or easy, especially in large teams or codebases.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#76

Post by carlmr »

twic wrote: post #57
fastball wrote: post #18

Earlier quoted context omitted.

.yml only makes sense in a word where every other extension is three letters. With variably length extensions, there isn't a good reason to not just use the full name.

Who cares? Just switch to .jsn and forget about it.

I'm sorry .jon would be more logical Javascript being one word and all.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#77

Post by fivefive55 »

john-radio wrote: post #62
Groxx wrote: post #11

First I've heard of "Taco Bell Programming"[1]. I love it, this needs to become a normal part of my lexicon. Also heck yes, it completely mirrors my experience - a few dozen tools combine in fairly simple ways to make absurdly useful results. The problem is finding the useful N-dozen elemental tools you'll use. [1]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. And with one dumb word choice, I suddenly can't share this otherwise good piece with most audiences.

The website it's linked on is called widgetsandshit.com. I think that's a bigger problem if you're that worried.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#78

Post by polygot »

robbya wrote: post #2

> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size. Some of the cloud providers have free hosting for public data sets (people who use the data incur cost to download/process the data). I'm not sure if this would qualify. * https://aws.amazon.com/opendata/public-datasets/ * https://azure.microsoft.com/en-us/servic…

I'm very interested in this data; if there is a torrent, I would be able to seed it if needed.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#79

Post by mhuffman »

john-radio wrote: post #62
Groxx wrote: post #11

First I've heard of "Taco Bell Programming"[1]. I love it, this needs to become a normal part of my lexicon. Also heck yes, it completely mirrors my experience - a few dozen tools combine in fairly simple ways to make absurdly useful results. The problem is finding the useful N-dozen elemental tools you'll use. [1]: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. And with one dumb word choice, I suddenly can't share this otherwise good piece with most audiences.

Which word? "manned"?

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#80

Post by Too »

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

zipapp, which comes bundled with python since 3.5, is your friend.

It can produce a single .pyz file which has all dependencies except the interpreter itself inside. Actually this format is compatible all the way back to Python 2.6, it's only the zipapp convenience scripts that are new.

https://docs.python.org/3/library/zipapp.html#creating-stand...

Post reply on HN