Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google has been DDoSing Sourcehut for over a year

drewdevault.com

Post reply on HN
Print view View original

101–110 of 225 posts

Re: Google has been DDoSing Sourcehut for over a year

#101

Post by nirvdrum »

verdverm wrote: post #91
ocdtrekkie wrote: post #74

Earlier quoted context omitted.

In this case, it's really hard to see thrashing other people's servers relentlessly to collect data you already have as anything but incredibly, incredibly poor engineering. Y'all should write him a check for that much resource waste.

It's less than $200 per month to send out 4G daily. If his business can't afford that, there is something else going on. What is the total daily bandwidth that sourcehut uses anyway? What percentage is go module fetching?

Plenty of bootstrapped businesses have better things to spend $200 / month on, let alone the time spent trying to figure out where all the anomalous traffic is coming from. As I understand it, it's not simple file fetches either. It's cloning a repo, which involves two-way communication, consumes CPU and RAM, and causes disk seeks. You're not slapping it on CloudFront and calling it a day. Finally, it looks to me like the costs are going to scale the more people he has using sourcehut and writing Go modules.

I don't really understand turning this around on him. Why should he have to subsidize Google? If it's not a problem, why do we have robots.txt at all? Just let bots hammer your site and cope with it.

The current situation can't be the optimal solution. It wasn't even present prior to Go 1.16. Only one company has the ability to change that. What should he do differently here? Why should he have to spend any of his time or money working around an issue he didn't create?

Re: Google has been DDoSing Sourcehut for over a year

#102

Post by verdverm »

ocdtrekkie wrote: post #95
verdverm wrote: post #91

Earlier quoted context omitted.

It's less than $200 per month to send out 4G daily. If his business can't afford that, there is something else going on. What is the total daily bandwidth that sourcehut uses anyway? What percentage is go module fetching?

That was a different user. The fact that a user not running a git hosting service is potentially eating $200 a month should queue you into the fact that the cost to Drew is likely drastically higher than that. Google should be sending reimbursement checks for the damage done here on this issue.

Drew is running a code hosting business and this is a cost of providing a feature to the users. He can pass the costs on if it is a problem. He has lots of options and his competitors are not making a big deal out of this.

I suspect he's drawn his line in the sand and wants to keep it going rather than finding a solution that works without requiring upstream changes.

Re: Google has been DDoSing Sourcehut for over a year

#103

Post by cycomanic »

joshuamorton wrote: post #98
cycomanic wrote: post #92

Earlier quoted context omitted.

So the solution is that everyone who does not want that behavior should request that they turn of refresh traffic for their domain? How long would it take until they would complain about people spamming their issue tracker if everyone who self-hosts a go module would start requesting this? Maybe they should request 500 times a day because that's what the proxy is doing.

> How long would it take until they would complain about people spamming their issue tracker if everyone who self-hosts a go module would start requesting this? This same question comes up for ~any CI service or any number of other services that regularly build from source. There's a reason that github maintains a cache and that repo clones are served from that instead of by invoking the git command on the server. A…

The guy who had 4G of traffic a day was self-hosting his go-module where he was the only user. Google downloaded his repo 500 times a day, that's some serious traffic for a single person, and no he was not a git hosting service.

Also traffic is not free, so even if they maintain a cache somewhere there is a cost to this.

Re: Google has been DDoSing Sourcehut for over a year

#104

Post by SSLy »

verdverm wrote: post #80
captainmuon wrote: post #43

It sucks having to work around something like this, but maybe the following would work: only allow the first checkout from a given Go node, blackhole later accesses. If the repository is modified or a certain time elapsed, reset and allow a download again. Also, if you want to escalate, I wonder if there is a way to create a fake git repository that expands to a huge amount of data when cloning, but uses minimal band…

Malicious behavior would come back to byte the author and his users & business

Yes, I also hope EU breaks Google apart.

Re: Google has been DDoSing Sourcehut for over a year

#105

Post by res0nat0r »

AlotOfReading wrote: post #93
johnduhart wrote: post #69

Earlier quoted context omitted.

> I mean, you didn't even consider implementing a simple fetch of an already cloned repository in your mirroring server code. So yeah, I'd argue that the bad faith part is actually justified. https://github.com/golang/go/issues/44577#issuecomment-11378... > We did consider caching clones, but it has security implications and adds complexity, so we decided not to. It is certainly not trivial to do and not something we…

Drew can often be very abrasive, but does it really matter in this case? His site is basically being DDoS'd. Yes, there are decent arguments why the golang infra doesn't cache or respect typical norms like robots.txt, but they don't change the unreasonableness of the underlying situation. Surely some mitigation could have been worked out in the year since the ticket was filed?

They offered to turn off refreshing of his domain it appears on Jun 8, 2021: https://github.com/golang/go/issues/44577#issuecomment-85692...

Re: Google has been DDoSing Sourcehut for over a year

#106

Post by joshuamorton »

cycomanic wrote: post #103
joshuamorton wrote: post #98

Earlier quoted context omitted.

> How long would it take until they would complain about people spamming their issue tracker if everyone who self-hosts a go module would start requesting this? This same question comes up for ~any CI service or any number of other services that regularly build from source. There's a reason that github maintains a cache and that repo clones are served from that instead of by invoking the git command on the server. A…

The guy who had 4G of traffic a day was self-hosting his go-module where he was the only user. Google downloaded his repo 500 times a day, that's some serious traffic for a single person, and no he was not a git hosting service. Also traffic is not free, so even if they maintain a cache somewhere there is a cost to this.

> The guy who had 4G of traffic a day was self-hosting his go-module where he was the only user. Google downloaded his repo 500 times a day, that's some serious traffic for a single person, and no he was not a git hosting service.

Ok so let me answer your previous question more explicitly:

> So the solution is that everyone who does not want that behavior should request that they turn of refresh traffic for their domain?

Yes.

> How long would it take until they would complain about people spamming their issue tracker if everyone who self-hosts a go module would start requesting this?

We have an answer to this. It is, empirically, a very long time, as there seem to be a handful (possibly only 2, certainly less than 5) users impacted negatively enough to report an issue. It does not usually make sense to solve an issue that doesn't exist. Presumably dozens of users reporting the same issue would cause a reprioritization, but if no one is complaining, there's no reason to change priorities.

Re: Google has been DDoSing Sourcehut for over a year

#107

Post by Beltalowda »

ocdtrekkie wrote: post #3

The fact that a programming language calls home to by Google by default should make it a non-starter for most sane developers. The fact that it calls home so it can DDoS other sites is low-key hilarious. And you'd think Google would know how to like... operate an efficient CDN, perhaps? Like, if this was managed by a competent company, you'd think this service would be akin to putting Cloudflare in front of your serv…

> The fact that a programming language calls home to by Google by default should make it a non-starter for most sane developers.

How is it any different from Rubygems, NPM, PyPi, or any other package repository? In most you can bypass it by using git repos, but almost no one does that. And the GOPROXY does offer real benefits, such as preventing left-pad problems.

As others have said, if you really think this is a huge problem for you then it's easy to disable, which is actually easier than what most other package managers offer, but I don't really see the harm in the first place.

Re: Google has been DDoSing Sourcehut for over a year

#108

Post by 8organicbits »

joshuamorton wrote: post #100
8organicbits wrote: post #64

Earlier quoted context omitted.

I think the problem with the proposed solution ( https://github.com/golang/go/issues/44577#issuecomment-85720... ) is that the refresh is used to keep the cache from getting too stale. Would it be reasonable for golang modules on sourcehub to be served from a stale cache? The problem is that the refresh does a full clone which is too heavyweight. The other user who accepted that proposed fix had a single module that…

If I'm reading correctly, it would not be stale, but be directly proxied, which would mean that a popular module could result in a higher overall qps (but this is also true if people clone directly from sourcehut). On the other hand, Drew's suggestion would actually result in stale caches. There comes a point (and it isn't a particularly far-off point) where disabling this would result in significantly increased traf…

> This may impact the freshness of your domain's data which users receive from our servers, since we need to have some caching on our end to prevent too frequent fetches.

They mention that requests still are served from the cache when refresh is disabled.

Re: Google has been DDoSing Sourcehut for over a year

#109

Post by minus7 »

captainmuon wrote: post #43

It sucks having to work around something like this, but maybe the following would work: only allow the first checkout from a given Go node, blackhole later accesses. If the repository is modified or a certain time elapsed, reset and allow a download again. Also, if you want to escalate, I wonder if there is a way to create a fake git repository that expands to a huge amount of data when cloning, but uses minimal band…

The problem is that a) you want it to work properly for users of Go modules hosted on your Git forge and b) you don't want to effectively waste time on working around a problem that the GoProxy engineers could just solve with a more intelligent (but maybe more complex) design.

Re: Google has been DDoSing Sourcehut for over a year

#110

Post by [deleted] »

ziml77 wrote: post #96
johnduhart wrote: post #69

Earlier quoted context omitted.

> I mean, you didn't even consider implementing a simple fetch of an already cloned repository in your mirroring server code. So yeah, I'd argue that the bad faith part is actually justified. https://github.com/golang/go/issues/44577#issuecomment-11378... > We did consider caching clones, but it has security implications and adds complexity, so we decided not to. It is certainly not trivial to do and not something we…

I don't particularly care for Drew, but the issue he's reported here seems totally valid. And if he requested that he be excluded from getting hit by the crawler, wouldn't that mean it would be impossible for people to use packages from sr.ht unless they change their config? Plus, it does seem reasonable to think that only one of the crawlers needs to hit the site. The global replication can happen at the FS level or…

[deleted]
Post reply on HN