Live data from Hacker News

Sourcehut will blacklist the Go module mirror

sourcehut.org

51–60 of 365 posts

Re: Sourcehut will blacklist the Go module mirror

#51
post #7

> and implies a trust relationship with Google to return authentic packages. The entire point of the sumdb (go.sum), is to prevent the need for such a relationship. If Google (or any proxy you use) tries to return questionable packages, it will be detected by that system.

In practice, I've never validated that any of the checksums in go.sum are either correct or consistent.

Re: Sourcehut will blacklist the Go module mirror

#52
post #49

From the GitHub issue, by a Googler: > For boring technical reasons, it would be a fair bit of extra work for us to read robots.txt […] This is coming from one of the biggest, richest, most well-staffed companies on the planet. It’s too much work for them to read a robots.txt file like the rest of the world (and plenty of one-man teams) do before hammering a server with terabytes of requests. If this is too much for…

Hm. https://sourcehut.org/robots.txt HTTP 404

presumably the robots.txt entry they are talking about is https://git.sr.ht/robots.txt

Re: Sourcehut will blacklist the Go module mirror

#53
post #35
post #8

I feel like Drew has been in a pissing match with the Go team for a while so this outcome doesnt surprise me. I love Go but not Google's stewardship of it. The tracking proxy, Russ' takeover / squash of the package management work, the weird silence / stonewalling on other community issues... Drew has a valid complaint. I hate to hear he was banned from the issue tracker but that sounds about right. As a sibling said…

Seems like they could also let one request per hour through, say, and then serve up the rest a 429. Users trying to clone their project would hit an almost certainly up to date Google cache and thus be happy and sr.ht save on pretty much all that traffic and thus be also happy.

I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many different places. So they would have to count requests from the specific user agent and by that point it is again special purpose work and load on sr.ht (whereas general rate limiting per IP might be a good idea anyway).

Re: Sourcehut will blacklist the Go module mirror

#54

> It should not be necessary to fetch the same git module up to 2,000 times per day. Holy cow Google! Wouldn't it behoove us to check if any changes occurred before downloading an entire repo?

could it be a CI service that builds the go-proxy server for testing and that build process does an initial clone of all sorts of go modules?

Re: Sourcehut will blacklist the Go module mirror

#55
post #53
post #35

Earlier quoted context omitted.

Seems like they could also let one request per hour through, say, and then serve up the rest a 429. Users trying to clone their project would hit an almost certainly up to date Google cache and thus be happy and sr.ht save on pretty much all that traffic and thus be also happy.

I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many different places. So they would have to count requests from the specific user agent and by that point it is again special purpose work and load on sr.ht (whereas general rate limiting per IP might be a good idea anyway).

Per blogpost:

> git clone requests with a GoModuleMirror User-Agent will receive a 429

So only one counter seems to be needed. Sure, it's doing work for an adversarial party, but it's not much extra work either.

Re: Sourcehut will blacklist the Go module mirror

#56
post #49

From the GitHub issue, by a Googler: > For boring technical reasons, it would be a fair bit of extra work for us to read robots.txt […] This is coming from one of the biggest, richest, most well-staffed companies on the planet. It’s too much work for them to read a robots.txt file like the rest of the world (and plenty of one-man teams) do before hammering a server with terabytes of requests. If this is too much for…

Hm. https://sourcehut.org/robots.txt HTTP 404

They're referring to the actual service domain not the public static domain.

https://git.sr.ht/robots.txt

Re: Sourcehut will blacklist the Go module mirror

#57
post #13
post #8

I feel like Drew has been in a pissing match with the Go team for a while so this outcome doesnt surprise me. I love Go but not Google's stewardship of it. The tracking proxy, Russ' takeover / squash of the package management work, the weird silence / stonewalling on other community issues... Drew has a valid complaint. I hate to hear he was banned from the issue tracker but that sounds about right. As a sibling said…

Have to agree in as much banning people with legitimate issues is unhelpful. I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels.

I've blocked many of the people that called for Peter's ban on Twitter. I don't want to be on their radar or a target of some sort of witch hunt. I consider myself a nice person and not inflammatory/offensive, but I'm a belt and suspenders type of person. It's more important that I can submit an issue on the tracker than interacting "socially" with people who have a higher chance of ostracizing targeted individuals. The risks in my mind now far outweigh the rewards.

Re: Sourcehut will blacklist the Go module mirror

#58
post #13

Earlier quoted context omitted.

Have to agree in as much banning people with legitimate issues is unhelpful. I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels.

> I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels. Bourgon was frequently helpful and great, but also frequently rude, condensing, dismissive, and generally just unpleasant. I've seen this countless of times first-hand on Slack, Reddit, and Lobsters. I specifically stopped interacting with him long before he was banned. Whether he's a g…

> And he also ranted about how people were all a bunch of morons. I don't blame anyone for getting tired of that.

Do you have a source?

Re: Sourcehut will blacklist the Go module mirror

#60
post #23

Earlier quoted context omitted.

What’s the deal on package mangement? They did quite a few iterations here so I don’t follow. Don’t know about the Russ takeover either

TL;DR version. Go package management was quite poor for many real use cases outside Google, so the community railed around many competing proposals, and when it appeared there was a clear winner, the Go team came out with their own solution instead. This after kind of supporting the ongoing community efforts.

> when it appeared there was a clear winner

Dep was very slow, tried to be very clever often resulting in it "cleverly" doing the wrong thing, and was generally a pain to deal with. At $dayjob we migrated to dep and then we had serious discussions if we should move back to glide (we didn't, as vgo was on the horizon by then). There were some plans to fix some of this IIRC, but they never really went beyond "plans".

It's always difficult if people spend a lot of time on a particular solution and then it turns out that's actually not what's desired, or if someone else thinks of an even better solution. All option suck here: chucking out people's work sucks (especially in open source volunteer-effort context), but accepting a bad solution "because people spent time on it" and then being stuck with that for years or decades to come sucks even more.

The communication could certainly have been better; a sort of "community committee" was given a repo on github.com/golang/dep and told "good luck". In hindsight the Go team should have watched development more closely and provided feedback sooner so that the direction could be adjusted. Actually, the entire setup probably wasn't a good idea in hindsight.

Post reply on HN