Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Laion Big Video Dataset

projects.laion.ai

Post reply on HN
Print view View original

1–10 of 28 posts

Re: Laion Big Video Dataset

#2

Post by topwalktown »

"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours."

I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themselves.

Re: Laion Big Video Dataset

#3

Post by ks2048 »

topwalktown wrote: post #2

"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours." I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themse…

I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html

They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.

Re: Laion Big Video Dataset

#4

Post by kevmo314 »

ks2048 wrote: post #3
topwalktown wrote: post #2

"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours." I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themse…

I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.

That’s a pretty penny on the bandwidth bill alone!

Re: Laion Big Video Dataset

#5

Post by kennyadam »

ks2048 wrote: post #3
topwalktown wrote: post #2

"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours." I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themse…

I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.

Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?

Re: Laion Big Video Dataset

#6

Post by m00dy »

kennyadam wrote: post #5
ks2048 wrote: post #3

Earlier quoted context omitted.

I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.

Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?

Nope, they’re different things. Proxy networks like Proxybase [0] use open-source clients and ask for the user’s consent before allowing them to join the network.

[0] https://proxybase.xyz

Re: Laion Big Video Dataset

#8

Post by vivzkestrel »

- can someone with expertise give us an overview of the architecture involved doing this

- let us say you ran yt-dlp inside python aiohttp

- surely your ll run a limit soon as your ip address will be flagged

- what solutions do we have to auto rotate proxies in python

- are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?

Re: Laion Big Video Dataset

#9

Post by faangguyindia »

vivzkestrel wrote: post #8

- can someone with expertise give us an overview of the architecture involved doing this - let us say you ran yt-dlp inside python aiohttp - surely your ll run a limit soon as your ip address will be flagged - what solutions do we have to auto rotate proxies in python - are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?

Google may supply cache boxes to ISPs which can be manipulated to let download any video you want

Re: Laion Big Video Dataset

#10

Post by MallocVoidstar »

kennyadam wrote: post #5
ks2048 wrote: post #3

Earlier quoted context omitted.

I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.

Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?

Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.
Post reply on HN