Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you store photos and videos?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 104 posts

Re: Ask HN: How do you store photos and videos?

#51

Post by Youden »

I don't trust any service with plaintext of my photos so I self host everything. I also have a relatively huge amount of data so most cloud storage is out even if I wanted it.

My setup is a Linux-based NAS with LUKS on everything and each HDD in ZFS with RAIDZ2. I store my photos in their own ZFS filesystem on the zpool and use a tool that automatically creates snapshots and keeps varying resolutions for various timeframes (e.g. 24 hours of hourly snapshots, a week of daily snapshots, a month of weekly snapshots and I think 6 months of monthly snapshots).

As far as organizing them goes, I put my photos in date-based folders (e.g. 2018-12-09) and for larger ones go through and give them more useful names like "2018-12-09 - Brother's birthday".

I tend to edit them with Darktable the most these days, which also stores XMP sidecars alongside the photos.

I haven't yet set up a backup, which I know is horrible. When I do I'll likely go with either Wasabi or a NUC or something in a nearby datacenter.

For remote access I have Owncloud, though I _really_ want something nicer. Unfortunately there isn't much that works well with RAW files.

Re: Ask HN: How do you store photos and videos?

#52

Post by LinuxBender »

xte wrote: post #50
LinuxBender wrote: post #48

Earlier quoted context omitted.

Certainly. The lftp mirror subsystem allows you to break up batches of files, or even a single file using p-get into multiple streams. Rsync can't do that. Rsync also doesn't work within a chroot environment, whereas lftp does. The exception is if you use native rsync, which means you are not encrypting the stream. You then get chroot, but not privacy. CPU overhead isn't really a limiting factor these days, assuming…

Hum, > The lftp mirror subsystem allows you to break up batches of files, or even a single file using p-get into multiple streams. I do not see any particular benefit from that... Network reliability is still a bit needed or you still have to restart... On restricting rsync on the other end there are many guide so again I do not see much benefit and that's not true that you can't encrypt the transfer otherwise with c…

Try lftp some time. There is a working example on the link I provided. It is much easier for people to set up than cryptcat / ncat / stunnel. I have found that people will follow the least path of resistance.

What I suspect you will find is that where a single rsync stream might reach a couple hundred mb/s, lftp can saturate the link if you wish. At least, that is what most people find that try it out.

Re: Ask HN: How do you store photos and videos?

#53

Post by Youden »

xte wrote: post #33
reacharavindh wrote: post #29

I have my photos and videos on a ZFS personal NAS at home, backed up independently to two external hard drives(secondary backup). The plan was to use recline to backup the ZFS NAS to Backblaze B2, but that bit is not setup yet. I have also been eyeing a project called photoprism to serve as a front end for my media, and perhaps Filebrowser as well as a secondary interface to view them as "files". https://github.com/p…

As a more fancy WebUI consider also GNU MediaGoblin :-)

Is there any way to point MediaGoblin at a folder on local disk and expose it as albums? I don't believe I've found one so far, it wants me to upload everything through it, which isn't going to happen.

Re: Ask HN: How do you store photos and videos?

#55

Post by j88439h84 »

Youden wrote: post #51

I don't trust any service with plaintext of my photos so I self host everything. I also have a relatively huge amount of data so most cloud storage is out even if I wanted it. My setup is a Linux-based NAS with LUKS on everything and each HDD in ZFS with RAIDZ2. I store my photos in their own ZFS filesystem on the zpool and use a tool that automatically creates snapshots and keeps varying resolutions for various time…

Syncthing has been good for me

Re: Ask HN: How do you store photos and videos?

#56

Post by jon_black »

I prefer to keep my data on my own machines. I have a freenas server and use syncthing to synchronise photos between my phone and the server, with a cron script to organise the photos into folders.

I wrote a blog post about the setup a couple of years ago: https://www.jonblack.me/using-freenas-and-syncthing-to-autom...

Re: Ask HN: How do you store photos and videos?

#57

Post by xte »

LinuxBender wrote: post #52
xte wrote: post #50

Earlier quoted context omitted.

Hum, > The lftp mirror subsystem allows you to break up batches of files, or even a single file using p-get into multiple streams. I do not see any particular benefit from that... Network reliability is still a bit needed or you still have to restart... On restricting rsync on the other end there are many guide so again I do not see much benefit and that's not true that you can't encrypt the transfer otherwise with c…

Try lftp some time. There is a working example on the link I provided. It is much easier for people to set up than cryptcat / ncat / stunnel. I have found that people will follow the least path of resistance. What I suspect you will find is that where a single rsync stream might reach a couple hundred mb/s, lftp can saturate the link if you wish. At least, that is what most people find that try it out.

I have used in the past, a distant past, both for ftp, sftp and ftps and sincerely I'm not much attracted...

On setup cryptcat and ncat require almost ZERO setup, and you still need to have ssh on the over side unless you are restricted by some common crappy webui so...

Also, I see no reason to saturate the link apart of try to milk more performance at the expense of all others, servers and network hardware are not there to be abused but to be used... It's the same story with people that abuse http range to start multiple downloads of a single big file trying to milk a bit of performance...

rsync performance are not much rsync related, they depend on network, encryption and compression you use, disk I/O, ... per se it's pretty efficient. Certainly if you can incremental send a zfs snap via mbuffer is far more efficient but that's far more efficient than any kind of logic file transfer...

Re: Ask HN: How do you store photos and videos?

#58

Post by JKCalhoun »

Looking through other user's replies, all the above for me. Everywhere I can back up photos I do.

iCloud primarily but all the old family photos I have scanned, retouched also are uploaded to Ancestry.com, Facebook....

These are, as I say, family photos (a couple photos are scans from tintypes — so some quite old). There are no privacy issue for me (quite the opposite, I want these shared with far flung relatives). So I try to copy them as many places as I can think to. Hoping of course that future generations of the family will still be able to find them somewhere.

Re: Ask HN: How do you store photos and videos?

#59

Post by xte »

Youden wrote: post #53
xte wrote: post #33

Earlier quoted context omitted.

As a more fancy WebUI consider also GNU MediaGoblin :-)

Is there any way to point MediaGoblin at a folder on local disk and expose it as albums? I don't believe I've found one so far, it wants me to upload everything through it, which isn't going to happen.

Hem no, it's about sharing multimedia contents... But since the OP cite many cloud services... That's at least decentralized and essentially FOSS.
Post reply on HN