Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sourcehut will blacklist the Go module mirror

sourcehut.org

Post reply on HN
Print view View original

271–280 of 365 posts

Re: Sourcehut will blacklist the Go module mirror

#271

Post by saurik »

throwaway894345 wrote: post #103
xena wrote: post #25

Earlier quoted context omitted.

Even more fun, Google doesn't even use Go modules internally! They use their own homegrown build system abomination that requires teams of people to maintain.

Pretty sure virtually every big tech company does the same thing, because they have to integrate a bunch of languages, target any arbitrary platform/architecture tuple, and support use cases like code generation and so on. There aren't any good solutions to this problem that don't require teams of people to maintain--the best in class open source solutions seem to be Bazel and Nix and these are beyond my skill to man…

Everyone ends up having to do this kind of stuff, and so the best tools are ones that take this in stride and make it easier to pull off even if it is at the expense of more pain for trivial project; and like, if you are going to try to build a baby version of the tool for beginners, you probably aren't the right person to even figure out what is required of that tool if you wouldn't normally use it.

Re: Sourcehut will blacklist the Go module mirror

#272

Post by kevincox »

cmatthias wrote: post #259
rsc wrote: post #209

The Go team has been making progress toward a complete fix to this problem. Go 1.19 added "go mod download -reuse", which lets it be told about the previous download result including the Git commit refs involved and their hashes. If the relevant parts of the server's advertised ref list is unchanged since the previous download, then the refresh will do nothing more than the ref list, which is very cheap. The proxy.go…

Why does the Go team and/or Google think that it's acceptable to not respect robots.txt and instead DDoS git repositories by default, unless they get put on a list of "special case[s] to disable background refreshes"? Why was the author of the post banned without notice from the Go issue tracker, removing what is apparently the only way to get on this list aside from emailing you directly? Do you, personally, find an…

FWIW I don't think this really fits into robots.txt. That file is mostly aimed at crawlers. Not for services loading specific URLs due to (sometimes indirect) user requests.

...but as a place that could hold a rate limit recommendation it would be nice since it appears that the Git protocol doesn't really have the equivalent of a Cache-Control header.

Re: Sourcehut will blacklist the Go module mirror

#273

Post by dcow »

tptacek wrote: post #247
dcow wrote: post #215

Earlier quoted context omitted.

I am no cheering fan, for sure, but I think it's disingenuous to say PGP is one of the least important systems on the internet. Debian package distribution, notably, depends rather pivotally on PGP to ensure authenticity. Keybase uses PGP as it's root trust mechanism. There are plenty of email services that use PGP to secure messages. I've even come across some recent (as in the last few years) startups using PGP to…

It is one thing to make a case for the continued maintenance of PGP, or even to say that it has a place in modern cryptography (that's an outré thing to say among cryptography engineers, but, whatever). It's another thing entirely to say that any cryptography engineer critical of PGP must have a weird personal vendetta against it, as you did upthread. Harsh criticism of the failings of PGP is practically an orthodoxy…

I didn't say "that any cryptography engineer critical of PGP must have a weird personal vendetta against it". I know the history and context around the matter. I know Filo has actually tried to do the work to replace PGP. I know it didn't stick. I imagine he more than many people understands how difficult the task of replacing it is. But in my opinion that should lead to a more tempered stance that represents an understanding of this subtlety. Instead we see him on team deprecate PGP software because it's not what We want golang users using. Excuse me if I attribute a small ounce of personal pride to that stance. I could be wrong. This is a discussion thread not a formal essay. I respect many things about Filo. I'm just critical of this particular crusade.

I mean yeah, you're right. PGP has been culturally deprecated for years now. There's no skirting that. I am quite happy that Debian is switching to minisign. Once that transition is complete that will be one less reason to keep PGP around. Really, I have absolutely zero allegiance to PGP. I'm just willing to admit that it works (and quite well) despite all the shortcomings that cryptography engineers love to spar with during happy hours. I sincerely do not disparage efforts to replace PGP. I am just tired of the passé mantra that PGP sux amirite or gtfo. As we both clearly understand, it's not really that simple.

Re: Sourcehut will blacklist the Go module mirror

#274

Post by cmatthias »

kevincox wrote: post #272
cmatthias wrote: post #259

Earlier quoted context omitted.

Why does the Go team and/or Google think that it's acceptable to not respect robots.txt and instead DDoS git repositories by default, unless they get put on a list of "special case[s] to disable background refreshes"? Why was the author of the post banned without notice from the Go issue tracker, removing what is apparently the only way to get on this list aside from emailing you directly? Do you, personally, find an…

FWIW I don't think this really fits into robots.txt. That file is mostly aimed at crawlers. Not for services loading specific URLs due to (sometimes indirect) user requests. ...but as a place that could hold a rate limit recommendation it would be nice since it appears that the Git protocol doesn't really have the equivalent of a Cache-Control header.

I'm taking the OP at his word here, but he specifically claims that the proxy service making these requests will also make requests independent of a `go get` or other user-initiated action, sometimes to the tune of a dozen repos at once and 2500 requests per hour. That sounds like a crawler to me, and even if you want to argue the semantic meaning of the word "crawler," I strongly feel that robots.txt is the best available solution to inform the system what its rate limit should be.

Re: Sourcehut will blacklist the Go module mirror

#275

Post by Arnavion »

counttheforks wrote: post #270
Arnavion wrote: post #232

Earlier quoted context omitted.

What does your ideal git host do when one of its (non-paying) clients uses bots to clones repos so hard that its (paying) human clients are unable to clone their repos?

Improve infrastructure. Apparently they are trivially DDOSable by anyone renting a few servers and running git clone in a loop? That's a problem they should solve. And no, blocking by user agent is not adequate protection.

Infrastructure costs. So you'd be willing to pay more for that?

Re: Sourcehut will blacklist the Go module mirror

#276

Post by [deleted] »

leetrout wrote: post #8

I feel like Drew has been in a pissing match with the Go team for a while so this outcome doesnt surprise me. I love Go but not Google's stewardship of it. The tracking proxy, Russ' takeover / squash of the package management work, the weird silence / stonewalling on other community issues... Drew has a valid complaint. I hate to hear he was banned from the issue tracker but that sounds about right. As a sibling said…

[deleted]

Re: Sourcehut will blacklist the Go module mirror

#277

Post by Arnavion »

kevincox wrote: post #272
cmatthias wrote: post #259

Earlier quoted context omitted.

Why does the Go team and/or Google think that it's acceptable to not respect robots.txt and instead DDoS git repositories by default, unless they get put on a list of "special case[s] to disable background refreshes"? Why was the author of the post banned without notice from the Go issue tracker, removing what is apparently the only way to get on this list aside from emailing you directly? Do you, personally, find an…

FWIW I don't think this really fits into robots.txt. That file is mostly aimed at crawlers. Not for services loading specific URLs due to (sometimes indirect) user requests. ...but as a place that could hold a rate limit recommendation it would be nice since it appears that the Git protocol doesn't really have the equivalent of a Cache-Control header.

As annoying as it is, there is precedent for this opinion with RSS aggregator websites like Feedly. They discover new feed URLs when their users add them, and then keep auto-refreshing them without further explicit user interaction. They don't respect robots.txt either.

Re: Sourcehut will blacklist the Go module mirror

#278

Post by josephcsible »

Beltalowda wrote: post #176
josephcsible wrote: post #142

Earlier quoted context omitted.

> Whether he's a great programmer/contributor not isn't really important here. Why not? Why shouldn't we offer more leeway to more valuable contributors?

The question is: how many other valuable contributors are you missing out on because of that person? The "classic" example of this is Ulrich Drepper, who maintained GNU libc for many years. Everyone agrees he's a great programmer. He's a better programmer than I am. But he was also ... difficult. More difficult than anyone else I've seen in a mainstream widely-used project. Many people didn't contribute purely becaus…

I don't think Ulrich Drepper is a good example. He wasn't just rude; he also blocked merging important changes. I think a better example is Linus Torvalds.

> In many cases you're not going to come out with better contributions and code.

I think that you would most of the time.

Re: Sourcehut will blacklist the Go module mirror

#279

Post by kevincox »

Arnavion wrote: post #277
kevincox wrote: post #272

Earlier quoted context omitted.

FWIW I don't think this really fits into robots.txt. That file is mostly aimed at crawlers. Not for services loading specific URLs due to (sometimes indirect) user requests. ...but as a place that could hold a rate limit recommendation it would be nice since it appears that the Git protocol doesn't really have the equivalent of a Cache-Control header.

As annoying as it is, there is precedent for this opinion with RSS aggregator websites like Feedly. They discover new feed URLs when their users add them, and then keep auto-refreshing them without further explicit user interaction. They don't respect robots.txt either.

I wouldn't expect or want an RSS aggregator to respect robots.txt for explicitly added feeds. That is effectively a human action asking for that feed to be monitored so robots.txt doesn't apply.

What would be good is respecting `Cache-Control`, which unfortunately many RSS clients don't, and just pick a schedule and poll on it.

Re: Sourcehut will blacklist the Go module mirror

#280

Post by kevincox »

cmatthias wrote: post #274
kevincox wrote: post #272

Earlier quoted context omitted.

FWIW I don't think this really fits into robots.txt. That file is mostly aimed at crawlers. Not for services loading specific URLs due to (sometimes indirect) user requests. ...but as a place that could hold a rate limit recommendation it would be nice since it appears that the Git protocol doesn't really have the equivalent of a Cache-Control header.

I'm taking the OP at his word here, but he specifically claims that the proxy service making these requests will also make requests independent of a `go get` or other user-initiated action, sometimes to the tune of a dozen repos at once and 2500 requests per hour. That sounds like a crawler to me, and even if you want to argue the semantic meaning of the word "crawler," I strongly feel that robots.txt is the best ava…

When I mean crawler I mean something that discovers new pages. Refreshing the same URL isn't really crawling.

But yes, it may be the best available solution in this case, even if I would argue that it isn't really it's main purpose.

Post reply on HN