I'm not a Go user so I don't really know how to Go proxy works, but if webmasters had a way to opt out via robots.txt or specific status codes (429) and then the proxy at Google sends a response to the client meaning "try it yourself" (i.e., try again in direct mode) then it could have been an adequate solution for this scenario.
Google has been DDoSing Sourcehut for over a year
211–220 of 225 posts
Re: Google has been DDoSing Sourcehut for over a year
#212Earlier quoted context omitted.
Considering they don't want caching, la what is considered basic rudimentary politeness, (and noting that the change increased traffic, so the proxy was intentionally doing busywork that no-one wanted or needed), The odds of them developing an unacceptable proxy were 100% Its akin to putting up an open exploitable DNS resolver in 2022 despite, with 2 seconds of research, the entire world telling you not to do that no…
Yeah so basically none of this is true? "Caching" is not considered basic rudimentary politeness, and what they did is not at all like putting up an "open exploitable DNS resolver" (also: putting up an "open exploitable DNS resolver" is pretty much still an industry norm). The "money generating machines" stuff doesn't add to the credibility of this argument. I get that you feel like you could design a better Go modul…
> "Caching" is not considered basic rudimentary politeness
You don't consider this a truth, I do. By google's standards it is a truth, https://cloud.google.com/web-risk/docs/caching for one example
> I get that you feel like you could design a better Go module proxy.
I add "sleep 1" at minimum to all my scraping loops, and refuse to use clients that don't keepalive and cache (if I'm hitting the same endpoint), I've done this always, so yes I could have designed a better proxy today, and a year ago, and two years ago, and three years ago, et, al.
> Money generating
They can pay people, I'm not doing it for free for a language I don't use
> if he took the Go project's offer up on just not having them clone modules hosted on his service so often.
Do you know that he hasn't tried? considering how well google responds to emails
Some people have principles that you might not understand, I too would not ask someone to not DDoS me, because I feel that is an absurd request.
So not a impact to the technical aspects but to the administrative aspects. People are paying devault to run a service as devault would, it would be improper for him to not run the service as expected of him.
> burn the extra bandwidth for a cacheing benefit that literally nobody on the Internet seems to be able to describe
bandwidth and CPU time both cost money
- - -
There are alternative approaches that have existed before 1.16, Deno with its pull once then pull only when told to approach, has not had a crisis of security, Every other language works on demand, NPM at most checks a dedicated security issues API designed for mass consumption. These things existed, none of them were referenced.
Re: Google has been DDoSing Sourcehut for over a year
#213Earlier quoted context omitted.
Yeah so basically none of this is true? "Caching" is not considered basic rudimentary politeness, and what they did is not at all like putting up an "open exploitable DNS resolver" (also: putting up an "open exploitable DNS resolver" is pretty much still an industry norm). The "money generating machines" stuff doesn't add to the credibility of this argument. I get that you feel like you could design a better Go modul…
> "Caching" is not considered basic rudimentary politeness You don't consider this a truth, I do. By google's standards it is a truth, https://cloud.google.com/web-risk/docs/caching for one example > I get that you feel like you could design a better Go module proxy. I add "sleep 1" at minimum to all my scraping loops, and refuse to use clients that don't keepalive and cache (if I'm hitting the same endpoint), I've d…
Do you know that he hasn't tried? considering how well google responds to emails
You can't reasonably build an argument by just making stuff up. Maybe, against all evidence, Google offered to stop hitting sr.ht with Go proxy cache refresh requests and then ignored DeVault when he took them up on it. But I'm not required to assume that very weird situation is what actually happened.
Re: Google has been DDoSing Sourcehut for over a year
#214Earlier quoted context omitted.
So far as I know, DeVault and the one other person who chimed in on his ticket are the only people who have ever availed themselves of this process, which may well have been created specifically for them. So, no, I don't think that's a valid point. Also: it is being fixed by the party responsible.
was being fixed. They just marked the issue as resolved and blocked any more comments on the thread.
They closed for comments because they started getting toxic, such is the modern internet mob mentality
Re: Google has been DDoSing Sourcehut for over a year
#215> I was banned from the Go issue tracker for mysterious reasons, so I cannot continue to nag them for a fix.¹ I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct). I tried to advocate for Linux distros to patch out GOPROXY by default, citing privacy reasons, but I was unsuccessful. I have no further re…
> In the meantime, if you would prefer, we can turn off all refresh traffic for your domain while we continue to improve this on our end. That would mean that the only traffic you would receive from us would be the result of a request directly from a user. This may impact the freshness of your domain's data which users receive from our servers, since we need to have some caching on our end to prevent too frequent fet…
> Disclosure: I was on the Go team at Google until earlier this month. Dealing with DeVault's bad faith arguments is one of the few things I won't miss of that job.
Speaking of bad faith arguments, aren't Google and its employees the ones that:
- Claimed that scrolling screenshot on Android are "infeasible" [INFEAS] despite Samsung, LG, HTC, etc. already implementing the feature in their forks/distributions of Android?
- Claimed "[t]he generic dilemma is this: do you want slow programmers, slow compilers and bloated binaries, or slow execution times?" as an excuse for not implementing Generics in Go?
- After years of claiming this nonsense finally implemented Generics in exactly those ways? [GoLang_GenImpl]
- Even when implementing Generics in exactly those old and previously known ways claim that "Russ Cox famously observed that generics require choosing among slow programmers, slow compilers, or slow execution times. We believe that this design permits different implementation choices." [GoLang_43651]
- "Loses" or "loses track" of their customer's mobile phones, and when said customers cancel the credit charges for the phones they never received or otherwise disappeared, block their Google accounts, including their access to email? [Google_Criminals]
Drew DeVault certainly deserves some or even a lot of criticism, but Google and its employees attacking other for "bad faith"? This should be acceptable in a civilized society.
[GoLang_43651]: https://go.googlesource.com/proposal/+/refs/heads/master/des...
[Google_Criminals]: For example: https://www.reddit.com/r/GooglePixel/comments/7nrx07/google_... and https://www.reddit.com/r/GooglePixel/comments/84sysx/update_... There have been other documented cases of the same so this is not a single uncharacteristic incident.
[GoLang_GenImpl]: https://go.googlesource.com/proposal/+/e0113ba8479092562cf9d...
[INFEAS]: https://issuetracker.google.com/issues/80491647#:~:text=Stat...
Re: Google has been DDoSing Sourcehut for over a year
#216Why not just stop using Go? Why not Nim? Crystal? why not even Haxe? Why not stop using things built by and built for EvilCorp (TM)?
Re: Google has been DDoSing Sourcehut for over a year
#217Earlier quoted context omitted.
I can't really speak for Drew's problem with the workaround, only my own from a design point of view - he may not even agree with my own issue at all for all I know, so I don't want to put words in his mouth. From my own perspective, however, the issue is about impact and responsibility. The excess traffic is their impact as it is caused by their design and is entirely of their choosing. Thus it becomes their respons…
It sounds like DeVault let them know their proxy was hitting his service too hard, and they immediately responded by saying they could dial it down. Seems like a pretty normal sequence of events? I feel like I have to be missing something here.
I can't speak for Devault, either, but I think that Sourcehut is intended to be selfhosted if you want. So it's probably too much pain to ask every language provider (might be a lot) to dial down the traffic. Imagine youre hosting modules from ~200 langs.
Re: Google has been DDoSing Sourcehut for over a year
#218Earlier quoted context omitted.
If the workaround didn't degrade service for Drew's go users, google would just implement it for all repositories.
How exactly does this workaround degrade service for DeVault's Go users? I'm not saying it doesn't; I'm saying nobody has given a simple explanation of what the impact would actually be, except for some people who have given clearly false explanations. It's OK to just not know, but if you don't know, it's weird to have strong opinions about it.
> How exactly does this workaround degrade service for DeVault's Go users?
I think you don't always get the latest stuff due to caching or so.
Re: Google has been DDoSing Sourcehut for over a year
#219If it were me, and I wasn't willing to just block the traffic, I might just set a 128 kbps limit on it and call it a day[1]. Eventually, the other side will figure out that their fetchers are all backed up and work out how to do their job without burning so much bandwidth. [1] Yeah, that can be a bit of a pain to setup depending on the server settings, but some people have to pay for bandwidth and server resources, s…
Keeping the rate-limited connection alive eats up resources. In this case, the organisation executing the DDOS attack has virtually limitless resources.
Re: Google has been DDoSing Sourcehut for over a year
#220Regardless of whether or not Drew is abrasive (I've never dealt with him so have no opinion), this is on Google . If a single service (Go's module crawler) is doing this to multiple third parties (and it is, by design) then requiring Drew to implement a workaround is the same thing as relying on every (relevant to the issue) third party to do the same thing. Which is bad design. This should be fixed at source or swit…
Can you help me understand the problem with the workaround, which is precisely to have Google's proxy not excessively poll DeVault's service? It really seems like DeVault's real argument here isn't about the impact of this on his service, but that he doesn't like the design of the proxy. He has a lot of standing to complain about impacts on his service, but essentially no standing to complain about designs he finds s…
The proxy _is_ pretty nasty design. Basically every time `golang` fetches modules, it uses Google's proxy out of the box.
This lets Google collect very granular data on who uses which modules, and also makes it opaque to webmasters who's using their services (it's all hidden behind google). It's honestly pretty nasty how Google has hidden this kind of "analytics" within a pretty dark grey area.
At the same time, the bot that handles the proxy seems to crawl everything, very often, and excessively, effectively DDOS'ing anyone hosting go modules.