91–100 of 422 posts
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
Earlier quoted context omitted.
"Raises the question of what we see is real" No they really don't, dishonest founders do that. You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies. I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their…
ive seen unscrupulous founders fake traction during diligence, which is my day job but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
From the article: > More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact. I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devi…
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
[0]: https://github.com/jow-/nlbwmon
[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…
At least anubis works for me. (I run umatrix) Unfortunately whatever HN is using routinely blocks my login with "Sorry." some websites just always give me 403.
I believe that's the HN application itself, not a WAF in front of it.
Earlier quoted context omitted.
ive seen unscrupulous founders fake traction during diligence, which is my day job but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
[flagged]
Can BitTorrent’s architecture contribute anything useful here? I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
We agree that it would be great if it was even more widely used.
Earlier quoted context omitted.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
Earlier quoted context omitted.
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware. If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing. With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 add…
by consent they mean a dialog/EULA with careful wording was display and the user clicked ok.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
