Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

11–20 of 216 posts

Re: Feed the bots

#11

Post by asgerhb »

akoboldfrying wrote: post #3

My initial reaction was that running something like this is still a loss, because it probably costs you as much or more than it costs them in terms of both network bytes and CPU. But then I realised two things: 1. If they are using residential IPs, each byte of network bandwidth is probably costing them a lot more than it's costing you. Win. 2. More importantly, if this became a thing that a large fraction of all web…

Not to mention they have to store the data after they download it. In theory storing garbage data is costly to them. However I have a nagging feeling that the attitude of these scrapers is they get paid the same amount per gigabyte whether it's nonsense or not.

Re: Feed the bots

#13

Post by delecti »

hyperhello wrote: post #12

Why not show them ads? Endless ads, with AI content in between them?

To what end? I imagine ad networks have pretty robust bot detection. I'd also be surprised if scrapers didn't have ad block functionality in their headless browsing.

Re: Feed the bots

#14

Post by CaptainOfCoit »

8organicbits wrote: post #7
CaptainOfCoit wrote: post #5

Earlier quoted context omitted.

> Can bot writers overcome this if they know the credentials? Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com instead of ht…

The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.

> but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier

If you're doing something alike to cracking then yeah. But if the credentials are right there on the landing page, and visible to the public, it's not really cracking anymore since you already know the right password before you try it, and the website that put up the basic auth is freely sharing the password, so you aren't really bypassing anything, just using the same access methods as everyone else.

Again, if you're stumbling upon basic auth and you try to crack them, I agree it's at least borderline illegal, but this was not the context in the parent comment.

Re: Feed the bots

#15

Post by CaptainOfCoit »

Filligree wrote: post #8
8organicbits wrote: post #7

Earlier quoted context omitted.

The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.

Sure, it’s a crime for the bots, but it would also be a crime for the ordinary users that you want to access the website. Or if you make it clear that they’re allowed, I’m not sure you can stop the bots then.

I don't think it'd be illegal for anyone.

The (theoretical) scenario is: There is a website (example.com) that publishes the correct credentials, and tells users to go to example.com/authenticate and put those there.

At no point is a user (or bot) bypassing anything that was meant to stop them, they're following what the website is telling them publicly.

Re: Feed the bots

#16

Post by DrewADesign »

8organicbits wrote: post #7
CaptainOfCoit wrote: post #5

Earlier quoted context omitted.

> Can bot writers overcome this if they know the credentials? Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com instead of ht…

The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.

The people in the mad dash to AGI are either driven by religious conviction, or pure nihilism. Nobody doing this seriously considers the law a valid impediment. They justify (earnestly or not) companies doing things like scraping independent artist’s bread and butter work to create commercial services that tank their market with garbage knockoffs by claiming we’re moving into a post-work society. Meanwhile, the US government is moving at a breakneck pace to dismantle the already insufficient safety nets we do have. None of them care. Ethical roadblocks seem to be a solved problem in tech, now.

Re: Feed the bots

#17

Post by OutOfHere »

The user's approach would work only if bots can accurately even be classified, but this is impossible. The end result is that the action is user's site is now nothing but markov garbage. Not only will bots desert it but humans will too.

Re: Feed the bots

#20

Post by comrade1234 »

I had to follow a link to see an example:

"A glass is not impossible to make the file and so deepen the original cut. Now heat a small spot on the glass, and a candle flame to a clear singing note.

— context_length = 2. The source material is a book on glassblowing."

Post reply on HN