Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Nearly Lost All of My Data

kevq.uk

Post reply on HN
Print view View original

231–240 of 262 posts

Re: I Nearly Lost All of My Data

#231

Post by fencepost »

shittyadmin wrote: post #134
fencepost wrote: post #130

Things that jumped out at me from this: First, the USB external is probably OK except that its USB circuitry has taken a power hit. If it's a standard SATA drive inside that could probably be shucked and accessed. Counter: Some of these have drives that are no longer SATA but have a bunch of the USB connectivity built into the drive. At that point, you'd probably be looking at a few hundred $ of data recovery costs (…

The fears of parity raid disk error rates are greatly overblown - this post does a good job of summing up the problem: http://www.raidtips.com/raid5-ure.aspx A bit of annecdata: I've been running raid5 and raid6 for years without issues on 4 and 8TB drives, scrubs come back successful every month despite claimed error rates, the drive deaths I've had have been sudden whole drive failures or write failures to a large…

The linked article is working through some of the numbers on a pure math basis but the real world is messy in ways that article doesn't consider.

Most notably, if a drive has failed there's a reason for it and a lot of the possible reasons will be shared with other drives in the array. Was there a manufacturing issue and all of the drives are from the same lot (pretty common). Is the RAID in a hostile environment (heat, vibration, bad power, etc.)? Heck, is the RAID normally very lightly used and going to have heat problems if it's under full load for hours (days?) during a rebuild?

There are also factors like how the RAID controller is going to handle another read error - will it drop a second disk if the drive reports a failure? For that matter, if an array drops to "degraded" due to an error are you going to immediately replace the drive or are you going to write it off as a one-time fluke and let the array rebuild? Do you keep a pool of spare drives around that you'll drop the failed one into after testing? I've regularly seen an array drop a drive due to something transient, then rebuild onto it and not have any more problems for years.

Even with a 2-drive failure in a RAID5 you're unlikely to lose much data - almost everything is likely still there on the disks unless there've been catastrophic failures (e.g. an array of the old "Deathstar" drives which were prone to head crashes). You just may need to do recovery which will generally mean imaging each of the drives and doing recovery based on working with those images instead of the original drives.

Re: I Nearly Lost All of My Data

#232

Post by ahnick »

_emacsomancer_ wrote: post #122
pstch wrote: post #121

Earlier quoted context omitted.

Encrypted ZFS with FreeBSD is not actually native ZFS encryption, it uses GELI to handle the encryption part (which may be why it's bad at handling power losses).

Ah, for some reason I was thinking that FreeBSD's implementation of OpenZFS had native encryption.

Hopefully someday. https://github.com/openzfs/openzfs/pull/489

Re: I Nearly Lost All of My Data

#233

Post by RealityVoid »

ValentineC wrote: post #230
RealityVoid wrote: post #194

Earlier quoted context omitted.

Yes, I was also thinking about perkeep and giving it a spin. I was considering a RAID 1 array for redundancy. Another thing I was considering was M-Disk[1], that can, supposedly, hold information for 1000 years. But I wonder what other people's experience with it is. Alternatively, I understand normal Blu-Ray disks should be able to have data retention of about 40 years, and that sounds decent to me. I was looking in…

> Thinking about it, I think this would be a great idea for a startup. Cheap data storage for long term storage, with competitive prices, the ability to per-pay, user-side encryption and a simple UI that grandma can use to drop photos. It should be able to guarantee that the information you desire to will outlive you. As a consumer, I wouldn't trust a startup for such needs, because there's a likelihood that the star…

> As a consumer, I wouldn't trust a startup for such needs, because there's a likelihood that the startup would either raise prices, pivot to a different service offering, or shut down entirely.

Fair enough. It's just that I think something in the field that can do these things would be filling a need. And all companies have to start _somewhere_

But I agree, getting a promise of long-term availability from a new company is a bit rich.

Thanks for the info about GCP - I looked mostly into AWS glacier and I know AWS did not have a prepay offer.

Re: I Nearly Lost All of My Data

#234

Post by [deleted] »

fencepost wrote: post #231
shittyadmin wrote: post #134

Earlier quoted context omitted.

The fears of parity raid disk error rates are greatly overblown - this post does a good job of summing up the problem: http://www.raidtips.com/raid5-ure.aspx A bit of annecdata: I've been running raid5 and raid6 for years without issues on 4 and 8TB drives, scrubs come back successful every month despite claimed error rates, the drive deaths I've had have been sudden whole drive failures or write failures to a large…

The linked article is working through some of the numbers on a pure math basis but the real world is messy in ways that article doesn't consider. Most notably, if a drive has failed there's a reason for it and a lot of the possible reasons will be shared with other drives in the array. Was there a manufacturing issue and all of the drives are from the same lot (pretty common). Is the RAID in a hostile environment (he…

[deleted]

Re: I Nearly Lost All of My Data

#235

Post by cptskippy »

bastawhiz wrote: post #227
cptskippy wrote: post #221

Earlier quoted context omitted.

> Bandwidth is cheap. Most ISPs in the US now have a 1TB datacap and charge ridiculous rates for overages.

Are you really pushing 1TB/mo, though? That would be a full quarter of OP's lifetime of data. And in those cases where you are, could you not afford business class internet for a bit of a premium?

3 Nest cameras and streaming in 4k (Netflix, YouTube, Amazon) use up quite a bit. Since I started tracking last April, 736gb a month has been our lowest. Most AAA video games are 40-80gb, and my partner and I both can work from home with always on VPN.

So downloading random Steam games to play or taking a few videos of the kids for family can easily result in an extra 150gb of incidental bandwidth.

Over Thanksgiving we hit 973gb with guests in the house. I reduced all the camera and Netflix quality settings the last week of November to avoid the penalty fee.

Re: I Nearly Lost All of My Data

#236

Post by Plaastix »

m3nu wrote: post #152
rsync wrote: post #135

Hard to believe we are 129 comments into this and there are zero mentions of 'borg' in the comment threads ... For those that don't know, borg is a backup utility[1] that has been called the "holy grail of backups"[2]. It takes your plaintext files and directories, chops them into gpg-encrypted chunks with encrypted, random filenames, and will upload (and maintain) them, with an efficient, changes-only update, to any…

Let's also mention Vorta, an open source GUI for Borg to make it as easy to use as commercial backup tools. We just started translating into different languages. All input and PRs are welcome. https://github.com/borgbase/vorta/ (I'm the original author, but Thomas, the current Borg maintainer is also very active.)

Thanks for sharing this! Vorta seems to be exactly what I was looking for. I can't seem to get it up and running on Mac OS.

Edit: It looks like Vorta doesn't obey Mac OS's Dark Mode so it looks like the app doesn't launch.

Re: I Nearly Lost All of My Data

#237

Post by Havoc »

chrsstrm wrote: post #19
Havoc wrote: post #15

After thinking about this kind of data apocalypse for a bit I realised that the portion of my data that is mission critical is actually really small. Throwing it on multiple clouds with version history isn't an issue. (I'd recommend an O365 sub + duplicati...in theory you can push like 5TB to MS cloud).

And here's your recurring reminder that all cloud storage services fingerprint your uploaded files looking for TOS violations. If you haven't encrypted your backups first you are at the will of the service you are using.

Yeah duplicati encrypts by default.

Plus ironically all the content that would score me a TOS violation doesn't get backed up since it's easily replaceable...

Re: I Nearly Lost All of My Data

#238

Post by blattimwind »

are595 wrote: post #224
blattimwind wrote: post #187

Earlier quoted context omitted.

The crypto in Borg is bad. Even has its own page in the docs about how bad it really is. The tl;dr is "there is one AES key per repository and we use this with AES-CTR and trust the remote server to not fiddle with the counter". https://borgbackup.readthedocs.io/en/stable/internals/securi... Last time I checked there was still no work being done on improving performance. Multithreading has been on the agenda for year…

I'm not sure that's a fair summary. It's very clearly stated that there is authentication performed after encryption. The MAC should prevent the remote server changing the counter or any encrypted bits.

> When the above attack model is extended to include multiple clients independently updating the same repository, then Borg fails to provide confidentiality

Re: I Nearly Lost All of My Data

#239

Post by phiresky »

ahnick wrote: post #138
rsync wrote: post #135

Hard to believe we are 129 comments into this and there are zero mentions of 'borg' in the comment threads ... For those that don't know, borg is a backup utility[1] that has been called the "holy grail of backups"[2]. It takes your plaintext files and directories, chops them into gpg-encrypted chunks with encrypted, random filenames, and will upload (and maintain) them, with an efficient, changes-only update, to any…

Have you checked out duplicacy[1]? It's support for incremental backups is fairly unparalleled in my experience. It does not have the same limitation of duplicity. [1] https://github.com/gilbertchen/duplicacy

Looks nice, but it is not open source.

Re: I Nearly Lost All of My Data

#240

Post by ahnick »

phiresky wrote: post #239
ahnick wrote: post #138

Earlier quoted context omitted.

Have you checked out duplicacy[1]? It's support for incremental backups is fairly unparalleled in my experience. It does not have the same limitation of duplicity. [1] https://github.com/gilbertchen/duplicacy

Looks nice, but it is not open source.

I know what you mean. It is not a normal open source license and for commercial use it requires a $20 a year subscription per user. With that said, I believe all the core source code is viewable and auditable if you use the CLI version, which is what I use on my FreeNas system and there is no charge for personal use. I think this may meet many people's personal backup requirements.
Post reply on HN