Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

121–130 of 216 posts

Re: Feed the bots

#121

Post by kelnos »

maurycyz wrote: post #116
kelnos wrote: post #111

Earlier quoted context omitted.

Interesting that babble.c doesn't compile (with gcc 14): babble.c: In function ‘main’: babble.c:651:40: error: passing argument 1 of ‘pthread_detach’ makes integer from pointer without a cast [-Wint-conversion] 651 | pthread_detach(&thread); | ^~~~~~~ | | | pthread_t * {aka long unsigned int *} In file included from babble.c:77: /usr/include/pthread.h:269:38: note: expected ‘pthread_t’ {aka ‘long unsigned int’} but a…

Sorry about that, stupid mistake on my side. I've fix the version on the server, an you can just edit the line to "pthread_detach(thread);" The snprintf() is only part of a status page, so you can remove it if you want. As for the threads, that could be an issue if directly exposed to the internet: All it would take for an attacker to open a whole a whole bunch of connections and never send anything to OOM the proces…

Not sure if I agree with you on the thread exhaustion issue. The client can still send a flood of correctly-formed requests; the reverse proxy will pass them all through. As I said above, yes, the fact that babble processes requests so quickly would make this harder, but you could still end up with (tens of?) thousands of concurrent requests if someone is really determined to mess with you.

A solution could be to limit concurrent requests in the reverse proxy, but personally I prefer to write software that doesn't require another piece of software, configured correctly, to keep it safe.

And regardless, even with ~25 years of C experience under my belt, I don't think I'd ever be wholly comfortable exposing my C code to the internet, even behind a reverse proxy. Not coming at you directly with this, but I'm frankly skeptical of anyone who is comfortable with that, especially for a one-off service that won't see a lot of use and won't get a lot of eyeballs on it. (And I'm especially uncomfortable with the idea of posting something like this on a website and encouraging others to use it, when readers may not understand the issues involved.)

Re: Feed the bots

#122

Post by rmunn »

lcnPylGDnU4H9OF wrote: post #64
CaptainOfCoit wrote: post #14

Earlier quoted context omitted.

> but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier If you're doing something alike to cracking then yeah. But if the credentials are right there on the landing page, and visible to the public, it's not really cracking anymore since you already know the right password before you try it, and the website that put up the basic auth is freely sharing the pa…

> freely sharing the password It doesn't have to be so free. It can be shared with the stipulation that it's not used in a bot. https://www.law.cornell.edu/uscode/text/17/1201 (a) Violations Regarding Circumvention of Technological Measures.— (1) (A) No person shall circumvent a technological measure that effectively controls access to a work protected under this title. This has been used by car manufacturers to deny…

If I was assigned the task of arguing that in court (though it would be really stupid to assign me, a non-lawyer, that task), I'd probably argue that it's not circumventing a locked door when you use the actual key in the lock; "circumventing" refers to picking the lock. It could still be unauthorized access if you stole the key, but that's a different thing than circumventing, and this law forbids circumventing.

Likewise, if the encryption key is sitting on disk next to the encrypted data, it's not "circumventing" the encryption to use that key. And if you handed me the disk without telling me "Oh, you're only allowed to use certain files on the disk" then it's fair to assume that I'm allowed to use all the files that you put on the disk before handing it to me, therefore not unauthorized access.

That argument might fail depending on what's in the EULA for the car's diagnostic software (which I haven't seen), but I feel it would be worth trying. Especially if you think you can get a sympathetic jury.

Re: Feed the bots

#123

Post by mcdeltat »

Maybe a dumb question but what exactly is wrong with banning the IPs? Even if the bots get more IPs over time, surely storing a list of bans is cheaper than serving content? Is the worry that the bots will eventually cycle through so many IP ranges that you end up blocking legit users?

Re: Feed the bots

#124

Post by gridspy »

kelnos wrote: post #121
maurycyz wrote: post #116

Earlier quoted context omitted.

Sorry about that, stupid mistake on my side. I've fix the version on the server, an you can just edit the line to "pthread_detach(thread);" The snprintf() is only part of a status page, so you can remove it if you want. As for the threads, that could be an issue if directly exposed to the internet: All it would take for an attacker to open a whole a whole bunch of connections and never send anything to OOM the proces…

Not sure if I agree with you on the thread exhaustion issue. The client can still send a flood of correctly-formed requests; the reverse proxy will pass them all through. As I said above, yes, the fact that babble processes requests so quickly would make this harder, but you could still end up with (tens of?) thousands of concurrent requests if someone is really determined to mess with you. A solution could be to lim…

Thread exhaustion attack

1. Start connections to a server

2. Hold connections open

3. Do nothing else

Server

1. Incoming connection. assign a thread.

2. Wait for request 3. Serve request

4. Close connection and thread / return to threadpool

Solution: Use a reverse proxy to handle the incoming connections. Typical reverse proxies such as nginx use event-based polling not a per-connection thread so they are immune to this issue.

Re: Feed the bots

#125

Post by hekkle »

andai wrote: post #119

I am confused where this traffic is coming from. OP says it's from well funded AI companies. But there are not such a large number of those? Why would they need to scrape the same pages over and over? Or is the scraping happening in real time due to the web search features in AI apps? (Cheaper to load the same page again than to cache it?)

On is website: https://maurycyz.com/projects/ai-tarpit/

He mentions that he had a "Chrome" browser send him 20 requests per second from the address: 43.134.189.59. If you look this address up on shodan.io you will see this address is for Tencent, a public company that makes AI, with an annual revenue of $92 Billion USD.

Re: Feed the bots

#126

Post by kalkin »

I don't think this robots.txt is valid:

  User-agent: Googlebot PetalBot Bingbot YandexBot Kagibot
  Disallow: /bomb/\*
  Disallow: /bomb
  Disallow: /babble/\*

  Sitemap: https://maurycyz.com/sitemap.xml
I think this is telling the bot named "Googlebot PetalBot Bingbot YandexBot Kagibot" - which doesn't exist - to not visit those URLs. All other bots are allowed to visit those URLs. User-Agent is supposed to be one per line, and there's no User-Agent * specified here.

So a much simpler solution than setting up a Markov generator might be for the site owner to just specify a valid robots.txt. It's not evident to me that bots which do crawl this site are in fact breaking any rules. I also suspect that Googlebot, being served the markov slop, will view this as spam. Meanwhile this incentives AI companies to build heuristics to detect this kind of thing rather than building rules-respecting crawlers.

Re: Feed the bots

#127

Post by _heimdall »

Macha wrote: post #9
8organicbits wrote: post #7

Earlier quoted context omitted.

The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.

The legal implications of torrenting giant ebook collections didn't seem to stop them, not sure why this would

Going back to Napster hasn't the gray area always been in downloading versus uploading?

If anyone could show that LLM companies have been uploading torrents then they really would be in trouble. If they are only proven to have downloaded torrents they're walking the line.

Re: Feed the bots

#128

Post by kelnos »

gridspy wrote: post #124
kelnos wrote: post #121

Earlier quoted context omitted.

Not sure if I agree with you on the thread exhaustion issue. The client can still send a flood of correctly-formed requests; the reverse proxy will pass them all through. As I said above, yes, the fact that babble processes requests so quickly would make this harder, but you could still end up with (tens of?) thousands of concurrent requests if someone is really determined to mess with you. A solution could be to lim…

Thread exhaustion attack 1. Start connections to a server 2. Hold connections open 3. Do nothing else Server 1. Incoming connection. assign a thread. 2. Wait for request 3. Serve request 4. Close connection and thread / return to threadpool Solution: Use a reverse proxy to handle the incoming connections. Typical reverse proxies such as nginx use event-based polling not a per-connection thread so they are immune to t…

You didn't actually address the concerns I laid out. And I acknowledged that a reverse proxy, appropriately configured, could mitigate the issue.

Re: Feed the bots

#129

Post by _heimdall »

lcnPylGDnU4H9OF wrote: post #64
CaptainOfCoit wrote: post #14

Earlier quoted context omitted.

> but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier If you're doing something alike to cracking then yeah. But if the credentials are right there on the landing page, and visible to the public, it's not really cracking anymore since you already know the right password before you try it, and the website that put up the basic auth is freely sharing the pa…

> freely sharing the password It doesn't have to be so free. It can be shared with the stipulation that it's not used in a bot. https://www.law.cornell.edu/uscode/text/17/1201 (a) Violations Regarding Circumvention of Technological Measures.— (1) (A) No person shall circumvent a technological measure that effectively controls access to a work protected under this title. This has been used by car manufacturers to deny…

How is this different than skipping the password and leaving the same terms of use for the content itself?

Re: Feed the bots

#130

Post by _heimdall »

lcnPylGDnU4H9OF wrote: post #67
peterlk wrote: post #45

Earlier quoted context omitted.

LLMs can now detect garbage much more cheaply than humans can. This might increase cost slightly for the companies that own the AIs, but it almost certainly will not result in hiring human reviewers

> LLMs can now detect garbage much more cheaply than humans can. Off the top of my head, I don't think this is true for training data. I could be wrong, but it seems very fallible to let GPT-5 be the source of ground truth for GPT-6.

I dotn think an LLM even can detect garbage during a training run. While training the system is only tasked with predicting the next token in the training set, it isn't trying to reason about the validity of the training set itself.
Post reply on HN