Sourcehut will blacklist the Go module mirror • Hacker News Board

Re: Sourcehut will blacklist the Go module mirror

#71

Post by psanford » Mon, Jan 09, 2023, 4:10 PM UTC

fluidcruft wrote: post #62

psanford wrote: post #43

Earlier quoted context omitted.

Why did sourcehut not take the offer to be added to the refresh exclusion list like the other two small hosting providers did? It seems like that would have resolved this issue last year.

I wondered that too, but then I wondered if that's what sourcehut have actually done. I didn't notice any details about how go module mirror will be blocked. Wouldn't the effect on sourcehut users be identical?

No. The Go team offered to add sourcehat to a list that would stop background refreshes. It would still allow fetches initiated by end users. The change sourcehat is making is to break end users unless they set some environment variables.

I've not seen any explanation about why the solution offered by the Go team was unacceptable. Its weird that that is completely left out of the blog post here.

Re: Sourcehut will blacklist the Go module mirror

#72

Post by jen20 » Mon, Jan 09, 2023, 4:10 PM UTC

xena wrote: post #25

pjmlp wrote: post #23

Earlier quoted context omitted.

TL;DR version. Go package management was quite poor for many real use cases outside Google, so the community railed around many competing proposals, and when it appeared there was a clear winner, the Go team came out with their own solution instead. This after kind of supporting the ongoing community efforts.

Even more fun, Google doesn't even use Go modules internally! They use their own homegrown build system abomination that requires teams of people to maintain.

Is this something other than Bazel (or the progenitor thereof)? If so, calling it an abomination compared to the unfinished go build system is rather strong.

Re: Sourcehut will blacklist the Go module mirror

#73

Post by mmis1000 » Mon, Jan 09, 2023, 4:11 PM UTC

winterqt wrote: post #16

pdonis wrote: post #10

Earlier quoted context omitted.

I don't understand why they have to do a fresh clone every time.

Or, as mentioned in the post, why they don't do a shallow clone if they have to fetch it every time for whatever reason. Seems like a weird decision either way.

Yep, a shallow clone is enough to get the latest version. And you can even filter the tree to make the download size even smaller given you only want the hash but not the contents (if the git server supports this feature)

A checkout with this can literally clone nothing but hash

    git clone --depth=1 --filter=tree:0  --no-checkout https://xxxx/repo.git
    cd repo
    git log

Re: Sourcehut will blacklist the Go module mirror

#74

Post by teeray » Mon, Jan 09, 2023, 4:12 PM UTC

ignaloidas wrote: post #44

teeray wrote: post #7

> and implies a trust relationship with Google to return authentic packages. The entire point of the sumdb (go.sum), is to prevent the need for such a relationship. If Google (or any proxy you use) tries to return questionable packages, it will be detected by that system.

If the proxy changes a new version of a package, when you update it, there's no way to detect it since it fetches through the cache anyways, so a poisoned sum will be added to sumdb, and anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them.

> anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them.

That is exactly the detection of a poisoned module in the ecosystem. It would break builds, issues would get filed, and a new version would be released (and the malicious party may not be so lucky this time since it’s trust on anyone’s first use).

Re: Sourcehut will blacklist the Go module mirror

#75

Post by [deleted] » Mon, Jan 09, 2023, 4:13 PM UTC

Semaphor wrote: post #58

Beltalowda wrote: post #46

Earlier quoted context omitted.

> I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels. Bourgon was frequently helpful and great, but also frequently rude, condensing, dismissive, and generally just unpleasant. I've seen this countless of times first-hand on Slack, Reddit, and Lobsters. I specifically stopped interacting with him long before he was banned. Whether he's a g…

> And he also ranted about how people were all a bunch of morons. I don't blame anyone for getting tired of that. Do you have a source?

[deleted]

Re: Sourcehut will blacklist the Go module mirror

#76

Post by simonw » Mon, Jan 09, 2023, 4:14 PM UTC

> Perform a shallow git clone rather than a full git clone; or, ideally, store the last seen commit hash for each reference and only fetch if it has been updated.

I'd be interested to understand why that solution hasn't been implemented yet.

Re: Sourcehut will blacklist the Go module mirror

#77

Post by Tomte » Mon, Jan 09, 2023, 4:14 PM UTC

psanford wrote: post #71

fluidcruft wrote: post #62

Earlier quoted context omitted.

I wondered that too, but then I wondered if that's what sourcehut have actually done. I didn't notice any details about how go module mirror will be blocked. Wouldn't the effect on sourcehut users be identical?

No. The Go team offered to add sourcehat to a list that would stop background refreshes. It would still allow fetches initiated by end users. The change sourcehat is making is to break end users unless they set some environment variables. I've not seen any explanation about why the solution offered by the Go team was unacceptable. Its weird that that is completely left out of the blog post here.

I think Google should DDoS noone. Not everyone until they opt out.

Re: Sourcehut will blacklist the Go module mirror

#78

Post by FiloSottile » Mon, Jan 09, 2023, 4:15 PM UTC

throwaway_62022 wrote: post #67

comex wrote: post #27

According to some comments in the linked GitHub issue, including [1] from last May, Drew could have simply asked to be excluded from automatic refresh traffic from the mirror. If I understand correctly, that would still leave traffic from the mirror when it’s acting as a direct proxy for someone’s request, but that is traffic that would be going to sr.ht regardless. For some reason he did not do this and instead chos…

I think it is problematic that we are using github issues as "support forum" for asking a git host provider to be excluded from the refresh list. This should not have come to that. Whatever happened to "reasonable defaults", so as a random person hosting a single Go module doesn't get DOSed - https://github.com/golang/go/issues/44577#issuecomment-86087... ?

Everyone can make their own assessment of what is a reasonable default and what counts as a DoS (and they are welcome to opt-out of any traffic), but note that 4GB per day is 0.3704 Mbps.

Re: Sourcehut will blacklist the Go module mirror

#79

Post by Beltalowda » Mon, Jan 09, 2023, 4:16 PM UTC

Semaphor wrote: post #58

Beltalowda wrote: post #46

Earlier quoted context omitted.

> I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels. Bourgon was frequently helpful and great, but also frequently rude, condensing, dismissive, and generally just unpleasant. I've seen this countless of times first-hand on Slack, Reddit, and Lobsters. I specifically stopped interacting with him long before he was banned. Whether he's a g…

> And he also ranted about how people were all a bunch of morons. I don't blame anyone for getting tired of that. Do you have a source?

https://github.com/golang/go/issues/30141#issuecomment-46427...

I don't have a full list of all posts at hand (some of which may be removed), but I've seen some other similar stuff as well; it's not an isolated incident. I was reading through the previous thread on this issue (goproxy sending loads of requests) and this one was posted as an example there.

Re: Sourcehut will blacklist the Go module mirror

#80

Post by throwaway894345 » Mon, Jan 09, 2023, 4:17 PM UTC

pjmlp wrote: post #21

leetrout wrote: post #8

I feel like Drew has been in a pissing match with the Go team for a while so this outcome doesnt surprise me. I love Go but not Google's stewardship of it. The tracking proxy, Russ' takeover / squash of the package management work, the weird silence / stonewalling on other community issues... Drew has a valid complaint. I hate to hear he was banned from the issue tracker but that sounds about right. As a sibling said…

Many issues with Go design steem from "how Google does it", this is just yet another one.

I doubt this is true in any objective sense. "issues" above is almost certainly subjective, and I don't think even in this particular case it's so much "how Google does it" as much as "Google isn't prioritizing limited Go-team resources to this issue right now".

In general, I've been very happy with Go's design for the last decade with relatively few, relatively minor exceptions. I specifically have enjoyed that the Go team has resisted demands from the wider community to add every feature that every other programming language has support for. Biasing toward minimalism has made the language very easy to learn and consequently any Go programmer can jump into any Go project and start being productive in a matter of minutes, and any non-Go programmer can be productive in a matter of hours. Similarly, the tooling is novel in that it has sane defaults (static, native compilation; testing out of the box; profiling out of the box; build tooling out of the box; reproducible package management and hosting out of the box; documentation generation and hosting out of the box; etc).