Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Please make this tool “opt-in” by default

github.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Please make this tool “opt-in” by default

#11

Post by xg15 »

version_five wrote: post #10

Ironically, the reason this tool is popular is because copyright prevents sharing of the datasets, and so all organizations like LAION can do is share URLs. The fuss people are kicking up about scrapers is nothing compared to what they'd do if their copyrighted images were in a public ML dataset. If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot o…

> being part of training data was fair use

It's not though, and never has been. Why should it be?

Why should people contribute to making their own jobs obsolete and not even be compensated for it?

> It would be win/win

In what way would this be a win for artists?

Re: Please make this tool “opt-in” by default

#12

Post by rolph »

version_five wrote: post #9
gus_massa wrote: post #8

Earlier quoted context omitted.

I guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider. We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other an…

The tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.

the tool also has settings that ignore opt out.

>> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.>>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<

Re: Please make this tool “opt-in” by default

#13

Post by rolph »

gus_massa wrote: post #8
wmf wrote: post #5

This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can…

I guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider. We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other an…

it seems like what would be reqired is to whitelist, known acceptable user agents, then review requests from green agents and whiten them based on X criteria, otherwise leave green agents green and put them in a hole.

Re: Please make this tool “opt-in” by default

#14

Post by version_five »

rolph wrote: post #12
version_five wrote: post #9

Earlier quoted context omitted.

The tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.

the tool also has settings that ignore opt out. >> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers. >>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<

Fair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool

Re: Please make this tool “opt-in” by default

#15

Post by scarface74 »

xg15 wrote: post #11
version_five wrote: post #10

Ironically, the reason this tool is popular is because copyright prevents sharing of the datasets, and so all organizations like LAION can do is share URLs. The fuss people are kicking up about scrapers is nothing compared to what they'd do if their copyrighted images were in a public ML dataset. If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot o…

> being part of training data was fair use It's not though, and never has been. Why should it be? Why should people contribute to making their own jobs obsolete and not even be compensated for it? > It would be win/win In what way would this be a win for artists?

> Why should people contribute to making their own jobs obsolete and not even be compensated for it?

Haven’t technologist been saying for years about blue collar workers whose jobs have been made obsolete by technology that they should learn a new skill?

Now that it’s hitting white collar workers, we are singing a different tune.

And most of the time especially here on HN most people want an expansive definition of fair use as long as it is directed toward using work created by the big media companies.

Re: Please make this tool “opt-in” by default

#16

Post by deanCommie »

While this may seem quaint, let's review the github package's README:

``` Opt-out directives

Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers.

To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'

See [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact) to understand better why you may decide to enable or disable this feature. ```

This does not pass the minimum thresholds of ethics for me. It may be legal, but it's not ethical. It's good that the default behaviour respects the headers. It's fine that there is an opt-out (if you didn't offer one, someone would just clone your code and offer it).

But if the most important information you provide is "here's how to use my tool unethically" rather than "here's why the default ethical use is important, though if you really have a good use case, here you go", then you're complicit in unethical use of this tool.

Re: Please make this tool “opt-in” by default

#17

Post by flangola7 »

xg15 wrote: post #11
version_five wrote: post #10

Ironically, the reason this tool is popular is because copyright prevents sharing of the datasets, and so all organizations like LAION can do is share URLs. The fuss people are kicking up about scrapers is nothing compared to what they'd do if their copyrighted images were in a public ML dataset. If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot o…

> being part of training data was fair use It's not though, and never has been. Why should it be? Why should people contribute to making their own jobs obsolete and not even be compensated for it? > It would be win/win In what way would this be a win for artists?

All of OpenAI's data collection is done under fair use

Re: Please make this tool “opt-in” by default

#18

Post by joshjob42 »

xg15 wrote: post #11
version_five wrote: post #10

Ironically, the reason this tool is popular is because copyright prevents sharing of the datasets, and so all organizations like LAION can do is share URLs. The fuss people are kicking up about scrapers is nothing compared to what they'd do if their copyrighted images were in a public ML dataset. If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot o…

> being part of training data was fair use It's not though, and never has been. Why should it be? Why should people contribute to making their own jobs obsolete and not even be compensated for it? > It would be win/win In what way would this be a win for artists?

It’s fair use because training sets are useless except for creating a new thing which transforms the inputs into something new. It’s not merely selling photos that already exist or something. Stable Diffusion has only about enough bits in the weights to represent a single pixel out of each image in the training set.

Re: Please make this tool “opt-in” by default

#20

Post by gus_massa »

version_five wrote: post #14
rolph wrote: post #12

Earlier quoted context omitted.

the tool also has settings that ignore opt out. >> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers. >>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<

Fair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool

There are two solutions

1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.

2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.

Post reply on HN