Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

51–60 of 687 posts

Re: Creepy Crawlies

#51

Post by daveguy »

jruohonen wrote: post #5
dingaling911 wrote: post #3

Earlier quoted context omitted.

looks like this https://github.com/rfonseca/xkcd-gnuplot

Thanks, and, yes, I don't write with LLMs, as seen above.

I would rather see some awkward phrasing than bland LLM slop!

Also, many plotting libraries include an xkcd style these days:

https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....

And libraries for various languages:

https://github.com/timqian/chart.xkcd

So if you have a preferred dev environment there's probably a way to set it to xkcd style.

Re: Creepy Crawlies

#52

Post by oowa »

chuckadams wrote: post #8

Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.

i think its only 7gb of data total, therefore there should be a strategy to fully optimize for this deterministic scenario, but instead of caching instead with maybe compiled streaming renderer(s)... but he said its ok for now.. and anyway why should they need to fix anything, the slurp is what sucks. pun intended

Re: Creepy Crawlies

#54

Post by jacobvosmaer »

easton wrote: post #37

Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)

Normal clones can reuse delta-compressed data the server stored on disk. Shallow clones impose a negative constraint: do not transfer data outside the requested commit depth. Pre-computed delta chains than contain unrequested data become unusable and the server must do delta compression on the fly to satsify the shallow clone.

Re: Creepy Crawlies

#55

Post by nxndbebdb »

nneonneo wrote: post #45

I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message). Distributed scrapers are g…

Almost all of my visits to cgit instances are through direct deep links. Hard to imagine someone randomly browsing git listings

Re: Creepy Crawlies

#56

Post by kardos »

edent wrote: post #43

WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have. If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?

Because they suspect that, sometimes, different content will be served by HTML vs alternate APIs

Re: Creepy Crawlies

#57

Post by NooneAtAll3 »

from what I see there are 2 solutions:

1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally

2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder

Re: Creepy Crawlies

#59

Post by nneonneo »

tptacek wrote: post #44

Tavis Ormandy called this, about Anubis, almost exactly a year ago: https://news.ycombinator.com/item?id=44962529 It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper…

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly.

The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.

If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.

Re: Creepy Crawlies

#60

Post by NooneAtAll3 »

tarpitt wrote: post #49

Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.

it's hard to separate spambot that only accesses 3-5 links per IP and a legit user. Changing content for legit user can be devastating
Post reply on HN