Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are people downloading YouTube videos server-side

news.ycombinator.com

Post reply on HN
Print view View original

1–6 of 6 posts

Ask HN: Are people downloading YouTube videos server-side

#1

Post by bigdatta »

I'm looking to download youtube videos so that I can transcribe them and then store the transcripts in my db. My application is a Node.js server that runs in a docker container on EC2. I'm using @distube/ytdl-core to download the Youtube video and it works perfectly on my local dev environment.

As soon I run it in AWS I get an error from Youtube that says "Sign in to confirm you’re not a bot". I'm guessing Youtube has blocked the IP's from AWS and forcing this, but there are a number of services I've seen that offer some kind of Youtube import (e.g. Opus Clip [1], Clay.ai [2], Veed.io [3] etc.), so my question is how are they getting around this restriction? The only possible solution I can think of is downloading the file locally and then uploading it but I feel like especially for Clay.ai that seems unlikely.

It seems like the suggestions online are to use a proxy server, but I don't trust random online proxy servers for a production application.

[1] https://www.opus.pro/ [2] https://www.clay.com/claybooks/get-youtube-transcripts-in-seconds [3] https://www.veed.io/

Re: Ask HN: Are people downloading YouTube videos server-side

#2

Post by PaulHoule »

Many organizations block IPs from places such as AWS, Azure, Tor and where it is trivial to get new addresses.

It's a great tactic for webcrawling but if you ever get caught they'll block those netblocks and nobody will get away with it again. (Try editing Wikipedia from Tor and see what happens.) I've been on both sides of that one.

Re: Ask HN: Are people downloading YouTube videos server-side

#3

Post by bigdatta »

PaulHoule wrote: post #2

Many organizations block IPs from places such as AWS, Azure, Tor and where it is trivial to get new addresses. It's a great tactic for webcrawling but if you ever get caught they'll block those netblocks and nobody will get away with it again. (Try editing Wikipedia from Tor and see what happens.) I've been on both sides of that one.

Are there suggestions for work arounds? I'm guessing it's a bit of a whack-a-mole problem, but it seems like others have figured out a solution for server-side downloads.

Re: Ask HN: Are people downloading YouTube videos server-side

#4

Post by PaulHoule »

bigdatta wrote: post #3
PaulHoule wrote: post #2

Many organizations block IPs from places such as AWS, Azure, Tor and where it is trivial to get new addresses. It's a great tactic for webcrawling but if you ever get caught they'll block those netblocks and nobody will get away with it again. (Try editing Wikipedia from Tor and see what happens.) I've been on both sides of that one.

Are there suggestions for work arounds? I'm guessing it's a bit of a whack-a-mole problem, but it seems like others have figured out a solution for server-side downloads.

Do your crawls from some place that isn't AWS. I bet there are many obscure data centers that aren't on their list.

I was brainstorming for answers to the paywall problem (e.g. system has to see what's behind the paywall) and one answer is a browser extension that can access everything the user's web browser can access (that user subscribes); in many cases this could be made pretty transparent.

Re: Ask HN: Are people downloading YouTube videos server-side

#5

Post by cindycindy »

PaulHoule wrote: post #4
bigdatta wrote: post #3

Earlier quoted context omitted.

Are there suggestions for work arounds? I'm guessing it's a bit of a whack-a-mole problem, but it seems like others have figured out a solution for server-side downloads.

Do your crawls from some place that isn't AWS. I bet there are many obscure data centers that aren't on their list. I was brainstorming for answers to the paywall problem (e.g. system has to see what's behind the paywall) and one answer is a browser extension that can access everything the user's web browser can access (that user subscribes); in many cases this could be made pretty transparent.

[flagged]
Post reply on HN