Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Processing 40TB of code from 10M projects with a dedicated server and Go

boyter.org

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#51

Post by skocznymroczny »

reubenmorais wrote: post #38
reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#52

Post by weberc2 »

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

Fully agree about Go; however, if you can't move away from Python, github.com/pantsbuild/pex is a pretty good alternative. It's an executable zip file that contains all of your python dependencies; you still need the Python interpreter and any shared objects that don't come with pip (e.g., for whatever reason, the Python graphviz package depends on the graphviz-devel system package and the latter doesn't ship via Pypi). It's still an order of magnitude worse than Go, but it's an order of magnitude better than pip or pipenv. :)

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#53

Post by folkrav »

theshrike79 wrote: post #46
reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous. The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera. Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

Why is it ridiculous? You'll need the related libraries either case, and by default virtualenv uses symlinks. The "full virtualenv" part makes it sound like it's heavy, or something like that.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#54

Post by twic »

EdwardDiego wrote: post #35
reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

Yeah, I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter. My initial thought was Python, but it needed a couple of 3rd party dependencies, and there wasn't an overly clean _and_ simple way to copy the script in from…

> I was writing a Docker image built around Prometheus' jmx_exporter intended to be used as a mixin for our Java apps, and part of what I needed to do was provide a simple script to preprocess separate config files to produce the config used by jmx_exporter

Did you consider, and stop me if this suggestion is completely wild, Java?

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#55

Post by twic »

skocznymroczny wrote: post #51
reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

I am an utter noob at Python packaging, but i would like to learn more, so excuse the basic question, but: why doesn't it work? And is there anything in the ecosystem which is like that, but does work?

I had heard that the Python packaging and deployment story had got a lot better in recent years, but this sounds like it still falls far short of table stakes.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#56

Post by folkrav »

eeZah7Ux wrote: post #39
reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

This is becoming a parody of software engineering. People are now irrationally scared of using dynamic libraries, OS packages and even directories, like virtualenvs. Instead, a simple solution is replaced with containers, or by rewriting tons of code in the new hyped language. Are we trying to create job security through unnecessary complexity?

Isolation is merely one utility of containers.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#57

Post by twic »

fastball wrote: post #18
ojkelly wrote: post #4

As much as I love the confirmation that .yml is the correct extension, I’m most amazed by the fact that there’s over a TRILLION lines of code public on GitHub. That’s an astronomical number. And that’s only what we can all see. Can’t imagine how much more is private. Also, it look’s like 20% of code is comments. Which feels just about right.

.yml only makes sense in a word where every other extension is three letters. With variably length extensions, there isn't a good reason to not just use the full name.

Who cares? Just switch to .jsn and forget about it.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#58

Post by weberc2 »

eeZah7Ux wrote: post #39
reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

This is becoming a parody of software engineering. People are now irrationally scared of using dynamic libraries, OS packages and even directories, like virtualenvs. Instead, a simple solution is replaced with containers, or by rewriting tons of code in the new hyped language. Are we trying to create job security through unnecessary complexity?

This is the weirdest straw man I've seen in a while. Python, not Go, benefits most from containerization. Installing "Dynamic libraries, OS packages, and even directories like virtualenvs" on target machines is far more complicated than sending a single file. I say all this as the DevOps guy in a Python shop.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#60

Post by weberc2 »

twic wrote: post #55
skocznymroczny wrote: post #51

Earlier quoted context omitted.

virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.

I am an utter noob at Python packaging, but i would like to learn more, so excuse the basic question, but: why doesn't it work? And is there anything in the ecosystem which is like that, but does work? I had heard that the Python packaging and deployment story had got a lot better in recent years, but this sounds like it still falls far short of table stakes.

I'm not sure why the Python community can't get it sorted out. We've been using Pipenv because it's one of a few tools that supports lockfiles/reproducible builds, and at the time we chose it, the Python packaging authority was advertising it as the official solution for Python packaging; however, after we were already invested in it, they backpedaled becaused the community realized it was very buggy and super slow (as in "it takes literally half an hour to add or remove or update any dependency").

We're finally getting around to migrating away, and we've settled on Twitter's github.com/pantsbuild/pants which is like Google's Bazel except that it's jankier in every way _except_ that Pants supports Python while Bazel doesn't (Bazel advertises Python support, but it doesn't work as the many Python 3 issues in their issue tracker can attest). The nicest thing about Pants is that it builds pex files (github.com/pantsbuild/pex) which are executable zip files that ship all of your dependencies except the Python interpreter and shared objects.

I'm still not very satisfied with this solution, and it's still far, far worse than Go's dependency management and packaging model, but it's a dramatic improvement over Pipenv, virtualenv, etc (gives us both reproducibility and performance).

Post reply on HN