Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Laion Big Video Dataset

projects.laion.ai

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Laion Big Video Dataset

#21

Post by vivzkestrel »

earthnail wrote: post #20
vivzkestrel wrote: post #8

- can someone with expertise give us an overview of the architecture involved doing this - let us say you ran yt-dlp inside python aiohttp - surely your ll run a limit soon as your ip address will be flagged - what solutions do we have to auto rotate proxies in python - are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?

Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.

- am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly

- are we talking here about using tor programmatically?

- what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3

- what kind of bot detection / evasion / ip address / proxy rotation mechanisms are needed

- are you familiar with libraries in the pypi ecosystem capable of tackling some of these concerns?

- is aiohttp / asyncio python enough for this paired with multiprocessing or do we have better options?

Re: Laion Big Video Dataset

#22

Post by m00dy »

earthnail wrote: post #20
vivzkestrel wrote: post #8

- can someone with expertise give us an overview of the architecture involved doing this - let us say you ran yt-dlp inside python aiohttp - surely your ll run a limit soon as your ip address will be flagged - what solutions do we have to auto rotate proxies in python - are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?

Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.

Dude, Tor's exit nodes are flagged everywhere.

Re: Laion Big Video Dataset

#23

Post by earthnail »

m00dy wrote: post #22
earthnail wrote: post #20

Earlier quoted context omitted.

Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.

Dude, Tor's exit nodes are flagged everywhere.

I downloaded the Laion Disco dataset this way.

Re: Laion Big Video Dataset

#24

Post by earthnail »

vivzkestrel wrote: post #21
earthnail wrote: post #20

Earlier quoted context omitted.

Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.

- am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly - are we talking here about using tor programmatically? - what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3 - what kind of bot detection / evasion / ip address / proxy rotation mechanisms…

- Use the python Library stem to control the Tor daemon and send sighup commands when necessary

- yt dlp has builtin support for tor

- write the retry logic using your favourite coding AI

- spin up a cheap cloud instance and let it run

Re: Laion Big Video Dataset

#25

Post by MallocVoidstar »

m00dy wrote: post #14
MallocVoidstar wrote: post #10

Earlier quoted context omitted.

Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.

No, they are not. Just ask any AI.

The companies say they have consent because saying "yeah we'll give you access to a botnet" is illegal.

https://cloud.google.com/blog/topics/threat-intelligence/dis...

>These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network.

If I install an app with a big ToS and buried in there is "we use third-party monetization SDKs" and buried in there is "welcome to our botnet" they do not actually have my consent.

Re: Laion Big Video Dataset

#26

Post by pia-nm »

m00dy wrote: post #22
earthnail wrote: post #20

Earlier quoted context omitted.

Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.

Dude, Tor's exit nodes are flagged everywhere.

[flagged]

Re: Laion Big Video Dataset

#27

Post by techsage »

Curious how useful very large video datasets are compared with smaller, better curated ones.

At some point, adding more videos probably helps less than improving the quality of the data.

Post reply on HN