Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google has been DDoSing Sourcehut for over a year

drewdevault.com

Post reply on HN
Print view View original

21–30 of 225 posts

Re: Google has been DDoSing Sourcehut for over a year

#21

Post by kevincox »

dmm wrote: post #4

> The Go team holds that this service is not a crawler, and thus they do not obey robots.txt This seems wrong. I guess I always assumed that robots.txt applied to non-humans.

I think they are right not to obey robots.txt in this case. If I tell Go to download a module it shouldn't follow robots.txt because I am a human and I requested it. This is similar to if you had private iCal URLs on your server the right think would be to deny them in robots.txt (in case a crawler found a leaked link) but a service that is monitoring a specific iCal calendar should still fetch it.

Basically robots.txt is more about what should be crawled than how. In this problem it appears that the traffic is desired in general, but it is being done far too often. robots.txt does have primative rate limiting configs but that seems to be a minor part of the file.

Of course like anything there is nuance and there is definitely some middle ground between crawlers and humans.

Re: Google has been DDoSing Sourcehut for over a year

#22

Post by toast0 »

ShowalkKama wrote: post #15
toast0 wrote: post #10

If it were me, and I wasn't willing to just block the traffic, I might just set a 128 kbps limit on it and call it a day[1]. Eventually, the other side will figure out that their fetchers are all backed up and work out how to do their job without burning so much bandwidth. [1] Yeah, that can be a bit of a pain to setup depending on the server settings, but some people have to pay for bandwidth and server resources, s…

Or you could mess with a random percentage of the requests: tarpit them, drop random packets, reply with malformed answers etc. If you keep the percentage low they might have a fun time debugging :)

If the rate limit is high enough, and pooled across all matching requests, you'll get different effects depending on the (global) concurrency, which might be tricky enough. And it might lead people into the right direction anyway.

Re: Google has been DDoSing Sourcehut for over a year

#23

Post by dgb23 »

sascha_sl wrote: post #5
ocdtrekkie wrote: post #3

The fact that a programming language calls home to by Google by default should make it a non-starter for most sane developers. The fact that it calls home so it can DDoS other sites is low-key hilarious. And you'd think Google would know how to like... operate an efficient CDN, perhaps? Like, if this was managed by a competent company, you'd think this service would be akin to putting Cloudflare in front of your serv…

Are you going to get upset at node for calling home to Microsoft (npm owned by github owned by microsoft) when using the supplied package management too?

The problems with npm are completely orthogonal to what the author is describing here.

Re: Google has been DDoSing Sourcehut for over a year

#24

Post by CryptoBanker »

toast0 wrote: post #18
mpol wrote: post #13

Earlier quoted context omitted.

I guess this is the professional response ;) I would just blacklist the IP addressess, just to see what would happen.

Yeah, but that's not an option according to the article. > I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct)

That’s more of a “won’t” than a “can’t”

Re: Google has been DDoSing Sourcehut for over a year

#25

Post by forty »

kevincox wrote: post #21
dmm wrote: post #4

> The Go team holds that this service is not a crawler, and thus they do not obey robots.txt This seems wrong. I guess I always assumed that robots.txt applied to non-humans.

I think they are right not to obey robots.txt in this case. If I tell Go to download a module it shouldn't follow robots.txt because I am a human and I requested it. This is similar to if you had private iCal URLs on your server the right think would be to deny them in robots.txt (in case a crawler found a leaked link) but a service that is monitoring a specific iCal calendar should still fetch it. Basically robots.t…

Precisely, here the service is fetching stuff regularly even if no one is asking anything, if I understand correctly.

Re: Google has been DDoSing Sourcehut for over a year

#26

Post by joshuamorton »

There is a very simple way to get them to stop sending the .5 qps that is described as a ddos.

The linked bug show another user successfully saying "please opt me out", and Google building the feature to do that in a week.

Drew has for some reason chosen not to ask for an opt out, even though it appears trivial and would probably be fixed by the weekend if he asked for it.

Re: Google has been DDoSing Sourcehut for over a year

#27

Post by eatonphil »

CryptoBanker wrote: post #24
toast0 wrote: post #18

Earlier quoted context omitted.

Yeah, but that's not an option according to the article. > I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct)

That’s more of a “won’t” than a “can’t”

It would mean that his host is not an option for any Go developers packaging libraries. So it's not really an option for their business, I'm guessing.

Re: Google has been DDoSing Sourcehut for over a year

#28

Post by tptacek »

ocdtrekkie wrote: post #3

The fact that a programming language calls home to by Google by default should make it a non-starter for most sane developers. The fact that it calls home so it can DDoS other sites is low-key hilarious. And you'd think Google would know how to like... operate an efficient CDN, perhaps? Like, if this was managed by a competent company, you'd think this service would be akin to putting Cloudflare in front of your serv…

You can phone anywhere you want; the proxy is configurable, and there are (from what I can see) several independent implementations of the proxy itself. Go defaults to Google's proxy; few people change the default because Google's proxy is very good.

Re: Google has been DDoSing Sourcehut for over a year

#29

Post by [deleted] »

threatofrain wrote: post #14

> I was banned from the Go issue tracker for mysterious reasons, so I cannot continue to nag them for a fix.¹ I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct). I tried to advocate for Linux distros to patch out GOPROXY by default, citing privacy reasons, but I was unsuccessful. I have no further re…

[deleted]

Re: Google has been DDoSing Sourcehut for over a year

#30

Post by dgb23 »

So this has been going on for a year, Drew DeVault has actively tried and explored several sane options and channels. That's a quite a bit of patience and goodwill before making a public blog post.

There are basic mechanisms in place both on the git side and on the protocol side that can prevent this kind of wasteful, digital harassment.

Post reply on HN