Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sourcehut will blacklist the Go module mirror

sourcehut.org

Post reply on HN
Print view View original

281–290 of 365 posts

Re: Sourcehut will blacklist the Go module mirror

#281

Post by ddevault »

tptacek wrote: post #249
ddevault wrote: post #239

Earlier quoted context omitted.

1. We would like to have a more complex discussion than saying "1" or "0" to this specific offer 2. This specific offer is not satisfactory: https://news.ycombinator.com/item?id=34313802

Clearly it was not satisfactory to you, since it was made over 8 months ago, and you didn't take them up on it. I'm objecting here only to the framing you've created that your ouster from the Go issue forum --- which we can see was done with cause --- is what precipitated this situation.

We can behave like adults, ask why it's not satisfactory, and come to a more agreeable mutual solution, or we can blithely offer an incomplete solution, muzzle the other party, and just continue our DDoS.

Re: Sourcehut will blacklist the Go module mirror

#282

Post by morelisp »

dcow wrote: post #269
morelisp wrote: post #252

Earlier quoted context omitted.

> rip it out of existence Come the fuck off this "mature personality" shit if you're going to write like this. He proposed freezing a module no one wanted to maintain in a library specifically meant to host stuff with weaker compat guarantees, he didn't hop in a DeLorean and kill Zimmermann's grandpa. Meanwhile, the critical project Drew insisted he keep it for is... deprecated and unmaintained!

[flagged]

[flagged]

Re: Sourcehut will blacklist the Go module mirror

#283

Post by rakoo »

kevincox wrote: post #272
cmatthias wrote: post #259

Earlier quoted context omitted.

Why does the Go team and/or Google think that it's acceptable to not respect robots.txt and instead DDoS git repositories by default, unless they get put on a list of "special case[s] to disable background refreshes"? Why was the author of the post banned without notice from the Go issue tracker, removing what is apparently the only way to get on this list aside from emailing you directly? Do you, personally, find an…

FWIW I don't think this really fits into robots.txt. That file is mostly aimed at crawlers. Not for services loading specific URLs due to (sometimes indirect) user requests. ...but as a place that could hold a rate limit recommendation it would be nice since it appears that the Git protocol doesn't really have the equivalent of a Cache-Control header.

> Not for services loading specific URLs due to (sometimes indirect) user requests.

A crawler has a list of resources it periodically checks to see if it changed, and if it did, indexes it for user requests.

Contrary to this totally-not-a-crawler, with its own database of existing resources, that periodically checks if anything changed, and if it did, caches content and builds chescksums.

Re: Sourcehut will blacklist the Go module mirror

#284

Post by Arnavion »

kevincox wrote: post #279
Arnavion wrote: post #277

Earlier quoted context omitted.

As annoying as it is, there is precedent for this opinion with RSS aggregator websites like Feedly. They discover new feed URLs when their users add them, and then keep auto-refreshing them without further explicit user interaction. They don't respect robots.txt either.

I wouldn't expect or want an RSS aggregator to respect robots.txt for explicitly added feeds. That is effectively a human action asking for that feed to be monitored so robots.txt doesn't apply. What would be good is respecting `Cache-Control`, which unfortunately many RSS clients don't, and just pick a schedule and poll on it.

robots.txt was originally created to include such bots. That they think they don't need to respect it goes against the original intent.

Eg: https://www.robotstxt.org/faq/kinds.html >"What's New" monitoring

Re: Sourcehut will blacklist the Go module mirror

#285

Post by cmatthias »

kevincox wrote: post #280
cmatthias wrote: post #274

Earlier quoted context omitted.

I'm taking the OP at his word here, but he specifically claims that the proxy service making these requests will also make requests independent of a `go get` or other user-initiated action, sometimes to the tune of a dozen repos at once and 2500 requests per hour. That sounds like a crawler to me, and even if you want to argue the semantic meaning of the word "crawler," I strongly feel that robots.txt is the best ava…

When I mean crawler I mean something that discovers new pages. Refreshing the same URL isn't really crawling. But yes, it may be the best available solution in this case, even if I would argue that it isn't really it's main purpose.

After reading this and your response to a sibling comment I wholeheartedly disagree with you on both the specific definition of the word crawler and what the "main purpose" of robots.txt is, but glad we can agree that Google should be doing more to respect rate limits :)

Re: Sourcehut will blacklist the Go module mirror

#286

Post by morelisp »

ignaloidas wrote: post #91
teeray wrote: post #74

Earlier quoted context omitted.

> anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them. That is exactly the detection of a poisoned module in the ecosystem. It would break builds, issues would get filed, and a new version would be released (and the malicious party may not be so lucky this time since it’s trust on anyone’s first use).

Considering how few people do so, I'm fairly certain it would take more than a month for somebody to catch that. But I guess it's also fairly easy to test it: just serve a slightly different version to the google's go mirror (by the user agent), and see how long until somebody complains to you about it.

> how few people do so

I think every company I know of with private Go modules (6-8 or so?) is running a module proxy, which will detect this. The several times we've detected this it's always been within 2-3 days of the upstream mistake. When I go to report a bug we're not always the first either.

Re: Sourcehut will blacklist the Go module mirror

#287

Post by typical182 »

ddevault wrote: post #257
rsc wrote: post #245

Earlier quoted context omitted.

This isn't true. Sourcehut reported a bug, and since the bug is somewhat involved to fix entirely, we asked what the impact of the bug is to them and offered to make a custom change for the site in the interim. The impact matters: the appropriate response is different for "I saw this in my logs and it looks weird but it's not bothering me" versus "this is causing serious problems for my site". We have been getting mi…

Your comment in this thread is the first time I've seen anyone mention that it was being worked on since... June 2021? This despite repeatedly raising the issue up until I was banned without explanation. I was never told, and still don't know, what disabling the refresh entails, the ban prevents me from discussing the matter further, and I was under the impression that no one was working on it. We have suffered a ser…

> the ban prevents me from discussing the matter further

Hi ddevault, FWIW, in May 2022 on that #44577 issue [0] you had opened, it looks like someone on the core Go team commented there [1] recommending that you email the golang-dev mailing list or email them directly.

Separately, it looks like in July 2022, in one of the issues tracking the new friendlier -reuse flag, there was a mention [2] of the #44577 issue you had opened. In the normal course, that would have triggered an automatic update on your #44577 issue... but I suspect because that #44577 issue had been locked by one of the community gardeners as "too heated", that automatic update didn't happen. (Edit: It looks like it was locked due to a series of rapid comments from people unrelated to Sourcehut, including about “scummy behavior”).

Of course, communication on large / sprawling open source projects is never quite perfect, but that's a little extra color...

[0] https://github.com/golang/go/issues/44577

[1] https://github.com/golang/go/issues/44577#issuecomment-11378...

[2] https://github.com/golang/go/issues/53644#issuecomment-11751...

Re: Sourcehut will blacklist the Go module mirror

#288

Post by Arnavion »

typical182 wrote: post #287
ddevault wrote: post #257

Earlier quoted context omitted.

Your comment in this thread is the first time I've seen anyone mention that it was being worked on since... June 2021? This despite repeatedly raising the issue up until I was banned without explanation. I was never told, and still don't know, what disabling the refresh entails, the ban prevents me from discussing the matter further, and I was under the impression that no one was working on it. We have suffered a ser…

> the ban prevents me from discussing the matter further Hi ddevault, FWIW, in May 2022 on that #44577 issue [0] you had opened, it looks like someone on the core Go team commented there [1] recommending that you email the golang-dev mailing list or email them directly. Separately, it looks like in July 2022, in one of the issues tracking the new friendlier -reuse flag, there was a mention [2] of the #44577 issue you…

The offer in [1] was to email the ML to ask for an exclusion, not to continue discussing the general issue which was still being discussed in the GH issue.

And given that they banned him for no reason, he is perfectly in the right to tell them that they should email him instead.

Re: Sourcehut will blacklist the Go module mirror

#289

Post by ddevault »

kevincox wrote: post #280
cmatthias wrote: post #274

Earlier quoted context omitted.

I'm taking the OP at his word here, but he specifically claims that the proxy service making these requests will also make requests independent of a `go get` or other user-initiated action, sometimes to the tune of a dozen repos at once and 2500 requests per hour. That sounds like a crawler to me, and even if you want to argue the semantic meaning of the word "crawler," I strongly feel that robots.txt is the best ava…

When I mean crawler I mean something that discovers new pages. Refreshing the same URL isn't really crawling. But yes, it may be the best available solution in this case, even if I would argue that it isn't really it's main purpose.

What you're thinking about, in my opinion, is best referred to as a spider.

Re: Sourcehut will blacklist the Go module mirror

#290

Post by ddevault »

Arnavion wrote: post #288
typical182 wrote: post #287

Earlier quoted context omitted.

> the ban prevents me from discussing the matter further Hi ddevault, FWIW, in May 2022 on that #44577 issue [0] you had opened, it looks like someone on the core Go team commented there [1] recommending that you email the golang-dev mailing list or email them directly. Separately, it looks like in July 2022, in one of the issues tracking the new friendlier -reuse flag, there was a mention [2] of the #44577 issue you…

The offer in [1] was to email the ML to ask for an exclusion, not to continue discussing the general issue which was still being discussed in the GH issue. And given that they banned him for no reason, he is perfectly in the right to tell them that they should email him instead.

Correct: it was made clear to me in no uncertain terms that the only thing I was allowed to say was "yes" or "no" to this offer.
Post reply on HN