Processing 40TB of code from 10M projects with a dedicated server and Go • Hacker News Board

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#41

Post by grumpydba » Tue, Oct 01, 2019, 9:29 AM UTC

eeZah7Ux wrote: post #39

reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

This is becoming a parody of software engineering. People are now irrationally scared of using dynamic libraries, OS packages and even directories, like virtualenvs. Instead, a simple solution is replaced with containers, or by rewriting tons of code in the new hyped language. Are we trying to create job security through unnecessary complexity?

> Are we trying to create job security through unnecessary complexity?

As a python/go dev, I can assure you that go is not much more complex. And as a server admin, it's much easier to deploy.

It does have drawbacks when compared to python, but it can often be the optimal solution.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#42

Post by gitgud » Tue, Oct 01, 2019, 9:30 AM UTC

One query that always interested me is how much duplication is in the Open-Source community. Like how many boiler-plate files are copied into projects.

It seems generating a `react/angular` project produces many files, that almost never get changed, so it would be interesting to know the most duplicated files...

I think this could also give valuable insight into how to make the language/frameworks better or simpler...

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#43

Post by throwaway77384 » Tue, Oct 01, 2019, 9:42 AM UTC

Glad to see Go Template make the list of languages with the most curse words in them.

NOT Go, but Go Template!! A subset of Go is causing more anguish than all of Python and only a sliver less than all of Rust.

Says it all really ;)

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#44

Post by giancarlostoro » Tue, Oct 01, 2019, 10:00 AM UTC

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

You didnt mention virtualenv. That is what I use. I even made a deb installer that forces itself to use virtualenv and fetches system python files for those packages available to Debian but not pip (kind of a hack but it worked! Look mah!). I agree though I like that with D and Go deployment seems simpler.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#45

Post by giancarlostoro » Tue, Oct 01, 2019, 10:05 AM UTC

grumpydba wrote: post #41

eeZah7Ux wrote: post #39

Earlier quoted context omitted.

This is becoming a parody of software engineering. People are now irrationally scared of using dynamic libraries, OS packages and even directories, like virtualenvs. Instead, a simple solution is replaced with containers, or by rewriting tons of code in the new hyped language. Are we trying to create job security through unnecessary complexity?

> Are we trying to create job security through unnecessary complexity? As a python/go dev, I can assure you that go is not much more complex. And as a server admin, it's much easier to deploy. It does have drawbacks when compared to python, but it can often be the optimal solution.

Also a Python dev. I do use VirtualEnv but the other thing is a huge performance boost for small performance intensive projects.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#46

Post by theshrike79 » Tue, Oct 01, 2019, 10:58 AM UTC

reubenmorais wrote: post #38

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous.

The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera.

Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#47

Post by mtberatwork » Tue, Oct 01, 2019, 11:05 AM UTC

theshrike79 wrote: post #46

reubenmorais wrote: post #38

Earlier quoted context omitted.

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous. The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera. Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

> The server might have the wrong Python version available

That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#48

Post by 08-15 » Tue, Oct 01, 2019, 11:19 AM UTC

Why would anyone store 1TB of data in highly redundant JSON?

Stored as protobuf, I estimate it would be 8x smaller. A custom binary format would be smaller again. Not only is that smaller, which saves storage and transfer cost, it's also proportionally faster to process.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#49

Post by tseabra » Tue, Oct 01, 2019, 12:37 PM UTC

mtberatwork wrote: post #47

theshrike79 wrote: post #46

Earlier quoted context omitted.

Installing a full virtualenv with all related libraries just to run one script is kind of ridiculous. The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera. Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.

> The server might have the wrong Python version available That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.

You'd need to install the right Python version though, and even with something like pyenv you might need to install system packages.

It really isn't as portable as one might wish.

Re: Processing 40TB of code from 10M projects with a dedicated server and Go

#50

Post by reacharavindh » Tue, Oct 01, 2019, 12:51 PM UTC

reubenmorais wrote: post #38

reacharavindh wrote: post #32

> I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging. Amen! This is why I am learning Go at the moment and considering using it instead of Python for admin and data processing tasks on a fleet of servers. The single binary deployment makes it a lot easier for users to…

I don't get it... Can't you just use a virtualenv? Then there's no worry about namespace pollution or conflicts between dependencies of different projects.

So a virtualenv for every script I want to run? each of that downloading a full Python interpreter and whatever modules that Python script uses? How is this any better than a statically compiled binary? Other than useless additional work and bloat on the servers?