Live data from Hacker News

Yark: Advanced and easy YouTube archiver now stable

github.com

131–140 of 148 posts

Re: Yark: Advanced and easy YouTube archiver now stable

#131
post #125

Does this have the ability to bet set to "grab highest available resolution" instead of specifying one? A lot of the material I'd like to archive has material from well before and after Youtube started supporting HD resolutions.

I would actually like the opposite: cap the resolution. I don't really need 4k videos.

Frankly I'd be ecstatic if I could limit it to 30fps versions as well. Aside from the occasional speedrun I have zero desire to increase the file size by 40 percent for nothing.

Re: Yark: Advanced and easy YouTube archiver now stable

#132

I have a youtube archiver script I'm using right now that I pulled from a thread on the data hoarder subreddit. My main issue is that I need the downloader to remove emojis from the filename because I sometimes sync them to Dropbox. Can this project do that?

yt-dlp has a `--restrict-filenames` flag

Re: Yark: Advanced and easy YouTube archiver now stable

#134
post #129

If you're interested in this kind of thing, you may also want to check out: - the Distributed YouTube Archive Discord: https://discord.com/invite/PQqks7eSKc - ArchiveTeam also do a significant amount of YT archiving: https://wiki.archiveteam.org/index.php/YouTube - a similar, but private effort: https://reddit.com/r/Archivists/comments/5uvfpw/youtube_arch...

I also archive many playlists with some code I wrote but I don't use a GUI.

https://github.com/chapmanjacobd/library/blob/f778e22bf80c58...

My focus is on error handling and trying to differentiate between unrecoverable errors and recoverable ones (try different proxy) but there's still a lot of work to be done.

Also look into https://github.com/swolegoal/squid-dl

Re: Yark: Advanced and easy YouTube archiver now stable

#135
post #103
post #87

Earlier quoted context omitted.

That's impressive indeed, but if we boil it down to just the things that are worth saving, removing duplicates, long and pointless livestreams, long videos that are just endless loops, etc., it could be done with much less space. If we also ignore auto generated spam[1] and harmful content (Elsagate, etc.), it would require even less. It's a nice thought experiment, but we really don't need to archive all of YT. That…

I would argue that backing up selectively instead of just grabbing everything makes the job harder, not easier. You might need less storage, but who will decide (and how) what gets stored and what doesn't?

Perhaps the backup algorithm needs to be coupled to the browser history of users. Stuff that isn't visited (or only for a few seconds) can be skipped.

Re: Yark: Advanced and easy YouTube archiver now stable

#136
post #134
post #129

If you're interested in this kind of thing, you may also want to check out: - the Distributed YouTube Archive Discord: https://discord.com/invite/PQqks7eSKc - ArchiveTeam also do a significant amount of YT archiving: https://wiki.archiveteam.org/index.php/YouTube - a similar, but private effort: https://reddit.com/r/Archivists/comments/5uvfpw/youtube_arch...

I also archive many playlists with some code I wrote but I don't use a GUI. https://github.com/chapmanjacobd/library/blob/f778e22bf80c58... My focus is on error handling and trying to differentiate between unrecoverable errors and recoverable ones (try different proxy) but there's still a lot of work to be done. Also look into https://github.com/swolegoal/squid-dl

Wow, I love the "daily tabs" concept. I'll install this and give it a go. Thanks!

"The use-case of tabs are websites that you know are going to change: subreddits, games, or tools that you want to use for a few minutes daily, weekly, monthly, quarterly, or yearly."

Re: Yark: Advanced and easy YouTube archiver now stable

#137

A while ago I did something similar. I'm already downloading various playlists via yt-dlp and wrote a web interface [1] to view/play/search them. My biggest annoyance at the time was importing my existing videos (and converting them to a streamable format, generating thumbnails and hover-previews etc). Do you have any plans of allowing to import existing yt-dlp folder (in the standard layout with a bunch of mkv files…

I do eventually, might be in v1.3 next month or sadly v1.4 depending on how much I've got on my plate :)

Re: Yark: Advanced and easy YouTube archiver now stable

#138

Does this have the ability to bet set to "grab highest available resolution" instead of specifying one? A lot of the material I'd like to archive has material from well before and after Youtube started supporting HD resolutions.

Looks like the "best" quality setting is hardcoded currently: https://github.com/Owez/yark/blob/676074ee3d9e379d15e52ffe2e... It would be nice to expose this setting via config file and/or the cli.

Working on some of the groundwork of it now in https://github.com/Owez/yark/pull/57. Higher quality might be a bigger issue to tackle because of dependence on ffmpeg but it'll be done in v1.3, slated for release in about a months time.

I'm going to cap videos to 1080p by default and have a config setting to customize this.

Re: Yark: Advanced and easy YouTube archiver now stable

#139

Earlier quoted context omitted.

With the rise of cheap 4k cameras, it goes faster than you think. I download guitar training videos where a 25 minute tutorial can be 2-3gbs. I believe those are “only” 2k resolution. If you do not download them in a space conscious format or recompress (using mildly aggressive settings can drop them to 100mb), a handful of videos can quickly fill a hard drive.

Why wouldn't one download something like that in 720P video/196kbps audio? I mean, there's a time and a place for 4K, but watching the zits on the face of a guy who tells you how to play C Am F G on a guitar isn't it. Not to mention, most of those cheap 4K cameras won't have optics to utilize those pixels; no quality will be lost in 720p.

My downloader defaults to the highest quality, so unless I remember to specify a setting, I get gigabytes of video for something that could easily be 480 quality. In the event I forget, I reencode with ffmpeg.

Re: Yark: Advanced and easy YouTube archiver now stable

#140

yt-dlp and a batch file that runs via Task Scheduler has been doing this for me for a couple of years now. I also grab the captions and throw that into a database so that I can search transcripts for a clip that I can remember but can't remember which video it's in. It was a fun weekend project.

I run mine with cron and it puts files in a special folder for plex: https://github.com/nburns/utilities/blob/master/youtube.fish Pulls from my watch later playlist which is quite handy

It looks like this depends on a "./add-video.py" script that isn't in the repository.
Post reply on HN