Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

131–140 of 216 posts

Re: Feed the bots

#131

Post by maurycyz »

kelnos wrote: post #121
maurycyz wrote: post #116

Earlier quoted context omitted.

Sorry about that, stupid mistake on my side. I've fix the version on the server, an you can just edit the line to "pthread_detach(thread);" The snprintf() is only part of a status page, so you can remove it if you want. As for the threads, that could be an issue if directly exposed to the internet: All it would take for an attacker to open a whole a whole bunch of connections and never send anything to OOM the proces…

Not sure if I agree with you on the thread exhaustion issue. The client can still send a flood of correctly-formed requests; the reverse proxy will pass them all through. As I said above, yes, the fact that babble processes requests so quickly would make this harder, but you could still end up with (tens of?) thousands of concurrent requests if someone is really determined to mess with you. A solution could be to lim…

> The client can still send a flood of correctly-formed requests

This is possible with any server. It's a known exploit and very difficult to fully mitigate: https://en.wikipedia.org/wiki/Denial-of-service_attack Whatever you do, they can always overwhelm your network connection.

And yes, there is inherent risk with exposing any service to the internet. That goes for any program, written in any language (remember Log4Shell?) doing any task.

Re: Feed the bots

#132

Post by _heimdall »

zkmon wrote: post #24

Really cool. Reminds me of farmers of some third world countries. Completely ignored by government, exploited by commission brokers, farmers now use all sorts of tricks, including coloring and faking their farm produce, without regard for health hazards to consumers. The city dwellers who thought they have gamed the system through high education, jobs and slick-talk, have to consume whatever is served to them by the…

What you describe sounds more like industrial farming than tricks played by third world farmers (whatever that means).

Industrial ag regularly treats product to modify the texture, color, and shelf life. Its extremely common to expose produce to various gases and chemicals to either delay or hasten ripening, for example. Other tricks are used while the plants are still in the ground or immediately after harvest, for example spraying grains with roundup to dry out more quickly.

Re: Feed the bots

#133

Post by _heimdall »

righthand wrote: post #65
Lord-Jobo wrote: post #62

Earlier quoted context omitted.

This is a very biased source discussing a very real prescription issue, and worth a glance for the statistics: https://www.farmkind.giving/the-small-farm-myth-debunked Tldr; the concept of farmers as small family farms has not been rooted in truth for a very long time in America

This is for livestock farming, I was specifically discussing agricultural farming. In general though, the easy rule of living and eating non-mega farmed food and sustainable living is to “eat aware”: My other advice is a one-size-fits-all food equation, which is, simply, to know where it came from. If you can't place it, trace it, or grow it/raise it/catch it yourself, don't eat it. Eat aware. Know your food. Don't w…

Are you proposing that eating industrially raised produce or meat is safer and healthier than alternatives?

Re: Feed the bots

#134

Post by maurycyz »

mcdeltat wrote: post #123

Maybe a dumb question but what exactly is wrong with banning the IPs? Even if the bots get more IPs over time, surely storing a list of bans is cheaper than serving content? Is the worry that the bots will eventually cycle through so many IP ranges that you end up blocking legit users?

It's often one IP (v4!) per one request. It's insane how many resources are being burned on this stupidity.

Part of the reason I did this is to get good numbers on how bad the problem is: A link maze is a great way to make otherwise very stealthy bots expose themselves.

Re: Feed the bots

#135

Post by [deleted] »

pavel_lishin wrote: post #98

The blog post ( https://maurycyz.com/misc/the_cost_of_trash/ ) says that gzip bombs don't work particularly well: > Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more. I thought a gzip bomb was crafted to explicitly be virtually unlimited in t…

[deleted]

Re: Feed the bots

#136

Post by maurycyz »

dmz73 wrote: post #100
pavel_lishin wrote: post #98

The blog post ( https://maurycyz.com/misc/the_cost_of_trash/ ) says that gzip bombs don't work particularly well: > Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more. I thought a gzip bomb was crafted to explicitly be virtually unlimited in t…

If the payload expands to something too large then it is easy to detect and ignore. Serve up thousands of 10kb or 100kb files that expand to 10s of MB with random garbage inside...possibly the same text but slightly modified. That will waste the time and CPU cycles and provide no value to them. Maybe also add a message you want to amplify so AI bots train on it.

The problem is that believable content doesn't compress well. You aren't going to get anywhere close to that 1:1000 compression ratio unless it's just a single word/character repeated thousands of times.

It's a choice between sending them some big files that will be filtered out long before they can do any real damage or sending them nonsense text that might actually make it's way into their training data.

Re: Feed the bots

#137

Post by maurycyz »

krzyk wrote: post #44

But why? Do they do any harm? They do provide source for material if users asks for it. (I frequently do because I don't trust them, so I check sources). You still need to pay for the traffic, and serving static content (like text on that website) is way less CPU/disk expensive than generating anything.

> Do they do any harm

Not to me, but I've known people who have had their sites DDoSed out of existence by the scrapers. On the internet, it's often the smallest sites with the smallest budgets that have the best content, and those are hit the worst.

> They do provide source for material if users asks for it

Not for material they trained on. Those sources are just google results for the question you asked. By nature, they cannot cite the information gathered by their crawlers.

> You still need to pay for the traffic

It's so little traffic my hosting provider doesn't bother billing me for it.

> and serving static content (like text on that website) is way less CPU/disk expensive than generating anything.

Sure, but it's the principle of the thing: I don't like when billion dollar companies steal my work, and then use it to make the internet a worse place by filling it with AI slop/spam. If I can make their lives harder and their product worse for virtually no cost, I will.

Re: Feed the bots

#138

Post by maurycyz »

vivzkestrel wrote: post #88

stupid question: why not encrypt your API response that only your frontend can decrypt. I understand very well that no client side encryption is secure and eventually once they get down to it, they ll figure out how this encryption scheme works but it ll keep 99% out won't it?

That would work, but I'd really prefer not to force users to run JavaScript, break RSS readers and slow down page loads (round trips are expensive). Adding a link maze to a random corner of the site doesn't impact users at all.

Re: Feed the bots

#139

Post by [deleted] »

gridspy wrote: post #124
kelnos wrote: post #121

Earlier quoted context omitted.

Not sure if I agree with you on the thread exhaustion issue. The client can still send a flood of correctly-formed requests; the reverse proxy will pass them all through. As I said above, yes, the fact that babble processes requests so quickly would make this harder, but you could still end up with (tens of?) thousands of concurrent requests if someone is really determined to mess with you. A solution could be to lim…

Thread exhaustion attack 1. Start connections to a server 2. Hold connections open 3. Do nothing else Server 1. Incoming connection. assign a thread. 2. Wait for request 3. Serve request 4. Close connection and thread / return to threadpool Solution: Use a reverse proxy to handle the incoming connections. Typical reverse proxies such as nginx use event-based polling not a per-connection thread so they are immune to t…

[deleted]

Re: Feed the bots

#140

Post by hekkle »

lcnPylGDnU4H9OF wrote: post #64
CaptainOfCoit wrote: post #14

Earlier quoted context omitted.

> but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier If you're doing something alike to cracking then yeah. But if the credentials are right there on the landing page, and visible to the public, it's not really cracking anymore since you already know the right password before you try it, and the website that put up the basic auth is freely sharing the pa…

> freely sharing the password It doesn't have to be so free. It can be shared with the stipulation that it's not used in a bot. https://www.law.cornell.edu/uscode/text/17/1201 (a) Violations Regarding Circumvention of Technological Measures.— (1) (A) No person shall circumvent a technological measure that effectively controls access to a work protected under this title. This has been used by car manufacturers to deny…

To be fair, even ignoring the Robots.txt is illegal in most western countries. I was a technical witness a while back, for a case about a bot ignoring the robots.txt. I said it was akin to a peeping tom ignoring a "no trespassing" sign, creeping into someones backyard, and looking through their window. Yes, they actually did bypass security controls, and therefore illegally "hacked" the site by ignoring it.
Post reply on HN