Live data from Hacker News

Sourcehut will blacklist the Go module mirror

sourcehut.org

131–140 of 365 posts

Re: Sourcehut will blacklist the Go module mirror

#131
post #10

Earlier quoted context omitted.

I don't understand why they have to do a fresh clone every time.

presumably the proxy backend is stateless

At the end of the day it has to store, consolidate, and present that state to the service's consumers, so there is state somewhere.

Re: Sourcehut will blacklist the Go module mirror

#132

Earlier quoted context omitted.

Everyone can make their own assessment of what is a reasonable default and what counts as a DoS (and they are welcome to opt-out of any traffic), but note that 4GB per day is 0.3704 Mbps.

Comes around $8-11 of egress monthly traffic on AWS. I would think twice before signing up for a service that charges me $10/month - not sure why this should be any different. Also, how do you opt-out? Imagine a random developer in a startup, running a Gitlab instance and then pushing a Go module there and only to be left with inexplicable traffic pattern(and bill). I have no skin in the game but this default _does n…

As an aside, I have done this and I do not have the problem. The most interesting question for me in all of this has always been, what triggers Google's request load?

Re: Sourcehut will blacklist the Go module mirror

#133

Earlier quoted context omitted.

I wondered that too, but then I wondered if that's what sourcehut have actually done. I didn't notice any details about how go module mirror will be blocked. Wouldn't the effect on sourcehut users be identical?

No. The Go team offered to add sourcehat to a list that would stop background refreshes. It would still allow fetches initiated by end users. The change sourcehat is making is to break end users unless they set some environment variables. I've not seen any explanation about why the solution offered by the Go team was unacceptable. Its weird that that is completely left out of the blog post here.

[flagged]

Re: Sourcehut will blacklist the Go module mirror

#134

Earlier quoted context omitted.

If the proxy changes a new version of a package, when you update it, there's no way to detect it since it fetches through the cache anyways, so a poisoned sum will be added to sumdb, and anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them.

> anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them. No, the error message you get is neutral about which side might be wrong - it says "verifying module: checksum mismatch" and "This download does NOT match the one reported by the checksum server." (I've seen it a lot because it also appears when module authors rebase, which a small but…

Wow, that is shocking. There is never a reason to rebase a public git repo, except maybe credentials leak in the past.

Re: Sourcehut will blacklist the Go module mirror

#135

Earlier quoted context omitted.

No. The Go team offered to add sourcehat to a list that would stop background refreshes. It would still allow fetches initiated by end users. The change sourcehat is making is to break end users unless they set some environment variables. I've not seen any explanation about why the solution offered by the Go team was unacceptable. Its weird that that is completely left out of the blog post here.

sourcehut's recommendations seem absolutely reasonable: (1) obey the robots.txt, (2) do bare clones instead of full clones, (3) maintain a local cache. I could build a system that did this in a week without any support from Google using existing open source tech. It's mind boggling that Google isn't honoring robots.txt, is requesting full clones, and isn't maintaining a local cache.

Despite the issue, I'm not convinced that Go doesn't do shallow fetches vs deep clones. Other issues (like Gentoo's issue with the proxy, I don't have a link handy sadly) point to fetches being done normally, not clones.

Re: Sourcehut will blacklist the Go module mirror

#136

From the GitHub issue, by a Googler: > For boring technical reasons, it would be a fair bit of extra work for us to read robots.txt […] This is coming from one of the biggest, richest, most well-staffed companies on the planet. It’s too much work for them to read a robots.txt file like the rest of the world (and plenty of one-man teams) do before hammering a server with terabytes of requests. If this is too much for…

I don't have any particular affinity for Google, but they're still a business and they're already developing the Go language (and relevant infrastructure) at their own expense. It's not like the Go team at Google has access to the entire Alphabet war chest like your "biggest, richest, well-staffed companies on the planet" suggests.

It seems like they're getting tons of bandwidth paid by the war chest if they don't care about this waste at all.

Re: Sourcehut will blacklist the Go module mirror

#137

> The situation remained so for over a year. In that time, I was banned from the Go issue tracker without explanation, and was unable to continue discussing the problem with Google Something tells me there's another side to this story that we aren't hearing.

Why? Google has quite a reputation for wrongfully banning people from things. This doesn't seem unusual for them at all.

Re: Sourcehut will blacklist the Go module mirror

#138
post #125

Earlier quoted context omitted.

> personal vendetta This is a weird definition of "personal", like PGP kicked his dog or something. The arguments he makes against it are detailed and the agreement of most working cryptographers, even if they don't agree with his specific deprecation schedule. Some people would call that "good engineering".

“Good engineering” would be to meticulously develop and standardize a replacement before idealistically purging the world of alleged “bad software”. Since this endeavor has yet to be undertaken, PGP it is. Good engineers understand this reality. Look, you can make solid arguments till you are blue in the face about why PGP is unclean and unfit for modern cryptography. And you can be 100% right. But that doesn't mean…

> develop and standardize a replacement

https://en.wikipedia.org/wiki/S/MIME

ok, I jest.

Re: Sourcehut will blacklist the Go module mirror

#139
post #15

Are google's requests done on-demand, or part of some scheduled refresh? I am wondering, because even if google's traffic is unreasonable, it might still be less than without the proxy. CI configurations are notoriously inefficient with dependency fetching, so I would not be surprised if the actual client traffic is massive and might overwhelm sourcehut if all migrate to direct fetches.

Per the issue, both.

I recently moved to docker build pipeline for a project, and it’s redownloading all deps on each source file change, unlike the efficient on disk incremental compilation, because of how docker layer caching works, so my usage skyrocketed (and my build times went from seconds to minutes).

Re: Sourcehut will blacklist the Go module mirror

#140

Earlier quoted context omitted.

> I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels. Bourgon was frequently helpful and great, but also frequently rude, condensing, dismissive, and generally just unpleasant. I've seen this countless of times first-hand on Slack, Reddit, and Lobsters. I specifically stopped interacting with him long before he was banned. Whether he's a g…

> Whether he's a great programmer/contributor not isn't really important here. I don't see why not. Personalities fall on a broad spectrum. Still seems strange to me that the recent broad pushes for more inclusiveness, including neuro-atypicality, does not cover people that inconvenience you personally. > that would chase people out of the community with his behaviour [...] but it did make the Go community a better p…

> but total exile is a big hammer that should be used more judiciously.

Well it was used judiciously, considering there are not more than couple of people banned in Go spaces in a decade.

Post reply on HN