Please make this tool “opt-in” by default

This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can satisfy everyone.

Re: Please make this tool “opt-in” by default

#6

Post by rolph » Sun, Apr 23, 2023, 10:55 PM UTC

wmf wrote: post #5

This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can…

ask first, or respect robots.txt no semantic malarchy

Re: Please make this tool “opt-in” by default

#7

Post by iudqnolq » Sun, Apr 23, 2023, 10:56 PM UTC

This tool seems designed to download from many domains at once based on the readme section about optimizing DNS resolution.

Considering that wouldn't a simple optimization be to shard by domain instead of arbitrarily?

Re: Please make this tool “opt-in” by default

#8

Post by gus_massa » Sun, Apr 23, 2023, 10:59 PM UTC

wmf wrote: post #5

This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can…

I guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider.

We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other anti abuse tool to everyone.

Re: Please make this tool “opt-in” by default

#9

Post by version_five » Sun, Apr 23, 2023, 11:05 PM UTC

gus_massa wrote: post #8

wmf wrote: post #5

This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can…

I guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider. We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other an…

The tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.

Re: Please make this tool “opt-in” by default

#10

Post by version_five » Sun, Apr 23, 2023, 11:11 PM UTC

Ironically, the reason this tool is popular is because copyright prevents sharing of the datasets, and so all organizations like LAION can do is share URLs. The fuss people are kicking up about scrapers is nothing compared to what they'd do if their copyrighted images were in a public ML dataset.

If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot of bandwidth, and make it more convenient for people working with these datasets. It would be win/win, but there's a prisoners dilemma + misplaced greed that means it will never happen like that.