Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Nearly Lost All of My Data

kevq.uk

Post reply on HN
Print view View original

121–130 of 262 posts

Re: I Nearly Lost All of My Data

#121

Post by pstch »

_emacsomancer_ wrote: post #114
pmlnr wrote: post #57

Earlier quoted context omitted.

Sure. Encrypted ZFS with FreeBSD, have a power loss, goodbye data. Should have read the manual though, it does tell you to make a backup of certain data ranges in case of encrypted ZFS for this specific case, so it's partly my fault. That said, I'm using ZFS ever since, but on top of LUKs with linux.

So LUKS is more resilient than native ZFS encryption to power loss?

Encrypted ZFS with FreeBSD is not actually native ZFS encryption, it uses GELI to handle the encryption part (which may be why it's bad at handling power losses).

Re: I Nearly Lost All of My Data

#122

Post by _emacsomancer_ »

pstch wrote: post #121
_emacsomancer_ wrote: post #114

Earlier quoted context omitted.

So LUKS is more resilient than native ZFS encryption to power loss?

Encrypted ZFS with FreeBSD is not actually native ZFS encryption, it uses GELI to handle the encryption part (which may be why it's bad at handling power losses).

Ah, for some reason I was thinking that FreeBSD's implementation of OpenZFS had native encryption.

Re: I Nearly Lost All of My Data

#123

Post by matthew-wegner »

ams6110 wrote: post #95

If you have 4TB of data saved at home you are a hoarder. Why do you have that much stuff. I don't have any data at home that I would miss very much if I lost it.

What? This is a wild assertion.

My photos alone are north of 4TB. That's DSLR, but not a crazy high-res one (to say nothing of people with video hobbies). I've always worked in small 2-8 person teams for companies that I've been heavily a part of, so that's a huge chunk too. But even discounting that data, I have quite a lot of projects that weigh in pretty heavily.

Yeah, I do have some datahoarding-type collections, because that's the sort of thing you end up doing when automation and total storage become commonplace, but just looking at "bytes I have created and will be lost forever if they vanish", I'm well north of 4TB. I think a lot of other people are too.

I don't mean this disparagingly, but if a person hasn't had any data-heavy hobbies, and has always been some kind of employee to a larger entity who manages data elsewhere, then yeah, your data footprint might be small. I imagine lots of HN regular types don't fit that mold, though.

(On the original link--I don't have much to add to the other comments here. But calling a 4-drive RAID5 setup robust in any sense is nuts. That's data loss waiting to happen, and probably made worse by thinking it is robust)

Re: I Nearly Lost All of My Data

#124

Post by clanrebornyes »

derekp7 wrote: post #13

The best solution I found as an alternative to an off-site copy is to set up a USB drive and something like a Raspberry Pi in my car. When I pull into the garage, the house server senses it and syncs new data to it. This can be supplemented further by having it auto sync to a computer in your office whenever you pull into work. Add in some monitoring so you can get a phone alert if any of the synced copies are more t…

I use syncthing for syncing when my raspberry pi joins a specific BSSID, sync starts.

It's susceptible to macspoofing but I hope my Pi won't join a spoofed network.

Re: I Nearly Lost All of My Data

#125

Post by ttmb »

There really isn't a story here, Kev. You blew a power supply on your Synology. You even said it yourself: "maybe this is a problem with the enclosures and the disks are fine".

If you'd put your disks into a replacement Synology unit, you would have been back online - config, data, and all - within a few minutes.

Re: I Nearly Lost All of My Data

#126

Post by BFLpL0QNek »

pmlnr wrote: post #57
bsder wrote: post #2

Z ... F ... S. And at least Raid-Z. Raid-5 and Raid-6 are now at probability of failure levels that your rebuild is likely to throw an error.

Sure. Encrypted ZFS with FreeBSD, have a power loss, goodbye data. Should have read the manual though, it does tell you to make a backup of certain data ranges in case of encrypted ZFS for this specific case, so it's partly my fault. That said, I'm using ZFS ever since, but on top of LUKs with linux.

Do you have more information on this?

I've just rebuilt my home server/nas going from 2x3tb disks in a mirror no encryption to 4 x 4tb disks in a striped mirror with native ZFS encryption on ZOL.

Most of my data is read only media content, 90% read use which is also extremely low io wise and write is only when adding new media. I was thinking power loss would only be an issue loosing data that is currently being written so would not corrupt the entire zpool?

If native ZFS encryption can lead to loosing a zpool on powerloss I might look in to buying a used cheap Dell r210 ii, stick two 8tb drives in as a mirror then stick it in a collocation data center and use native zfs send/recv incremental snapshots for offsite. Looks to be cheaper then rsync.net $30 per TB/month for ZFS when you got 2-3TB+, also can do the initial sync locally. Still looking at $700+ a year after initial hardware costs.

Re: I Nearly Lost All of My Data

#127

Post by clanrebornyes »

gtirloni wrote: post #24
derekp7 wrote: post #13

The best solution I found as an alternative to an off-site copy is to set up a USB drive and something like a Raspberry Pi in my car. When I pull into the garage, the house server senses it and syncs new data to it. This can be supplemented further by having it auto sync to a computer in your office whenever you pull into work. Add in some monitoring so you can get a phone alert if any of the synced copies are more t…

Does the RaspberryPi have enough CPU for disk encryption?

I am trying to find cheaper device between NUC and Pi.

Lot fewer choices are available.

NUC is expensive. I wonder when will 8core ARM based NUC will arrive in the market.

No idea is NUC can be powered by Xiomi Mi 2i powerbank but a Raspberry Pi can be

Priorities:

1. Gigabyte ethernet

2. Small form factor NUC or smaller.

3. 8+ cores

4. Very low power consumption

Re: I Nearly Lost All of My Data

#129

Post by ValentineC »

setquk wrote: post #28

Many years ago I watched someone stuff the DVDRW (remember them?) which contained all his stuff into a work PC’s pioneer slot loader drive to get some music off it. We stood there and the drive went bzzt, clang, bzzt, clang then sped up way faster than it was supposed to go. This was followed by a large bang and bits of DVDRW coming flying out and then a crunching noise. From this I learned about single points of fai…

> A plain file system you can just copy your shit back from is the closest thing to a guarantee.

These days, I'd suggest using a file system with some form of file checksum metadata. If one values the integrity of their data, bit rot is a thing.

Re: I Nearly Lost All of My Data

#130

Post by fencepost »

Things that jumped out at me from this:

First, the USB external is probably OK except that its USB circuitry has taken a power hit. If it's a standard SATA drive inside that could probably be shucked and accessed. Counter: Some of these have drives that are no longer SATA but have a bunch of the USB connectivity built into the drive. At that point, you'd probably be looking at a few hundred $ of data recovery costs (yes, that little). Professional recovery of the RAID would be more expensive because pricing is often based on the number and capacity of the drives.

Second, RAID5? I know these were only 1TB drives, but be very wary of anything with only a single parity disk if you're looking at drives of 1TB or larger, particularly if they're sequentially-numbered drives from the same lot. With modern TB+ drives there's a not-insignificant chance of drive errors as you hammer the remaining drives to rebuild the array. If building one of these now, the price difference between a RAID5 of smaller disks and a RAID6 of larger ones is probably only a few dollars.

Third, if actually doing recovery the first thing you want to do is image the drives and work from the images. ddrescue is probably your simplest option there, but yes you're going to need a big chunk of drive space available.

Post reply on HN