Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FreeNAS: Open Source Storage Operating System

freenas.org

Post reply on HN
Print view View original

171–180 of 182 posts

Re: FreeNAS: Open Source Storage Operating System

#171

Post by BrainInAJar »

feld wrote: post #119
NeutronBoy wrote: post #108

Earlier quoted context omitted.

I used ZFS in two production instances a few years ago on Solaris, and had two instances of a bug that totally corrupted the superblock on each drive, meaning that all data was gone with no way to recover. Now I'm back to HW RAID, LVM, and ext4, which works a treat, and I'll be using that until I have more faith in ZFS.

You had incredibly bad luck then unless you weren't running Solaris on Sun hardware, in which case you may have shot yourself in the foot with a poor disk controller and/or no ECC.

Sun hardware is expensive junk, but a bad disk controller that lies to ZFS about sync status is definitely in the risk model.

Re: FreeNAS: Open Source Storage Operating System

#172

Post by cbsmith »

baruch wrote: post #58
cbsmith wrote: post #41

Earlier quoted context omitted.

So many levels to this. Drives themselves remap sectors to handle bad sectors, and some have integrated ECC bits... there are also things at the sensor level to help recover from errors. Once you are at the block level, you have simple parity checking with some redundancy models, but the good ones use ECC bits. Usually this is done at the block device layer, but ZFS and some other filesystems do this at the filesyste…

An HDD/SSD without ECC will not work, and wouldn't have worked for a long time now. Many of the reads performed will require some ECC use. ECC is always written alongside the actual block and the overhead for ECC is the reason for the move from 512b sectors to 4Kb sectors in HDDs. For SSDs the data is already written in different block sizes depending on the NAND and the internal representation and ECC is done for la…

> An HDD/SSD without ECC will not work, and wouldn't have worked for a long time now. Many of the reads performed will require some ECC use.

Yeah, I expressed that badly. I always forget about the ECC bytes that are in the firmware.

> The probability of failure during rebuild is not really directly linked to drive size, the usual interpretation of the drive BER is wrong (media BER is stated across a large population of drives rather than just one drive).

Regardless of interpretations of BER, I can't agree about drive sizes. The phenomenon of failures during rebuild is well documented and the driving principle behind double-parity RAID. Adam Leventhal (who ought to know this stuff better than either of us) wrote a paper several years back on the need for triple-parity RAID, and it was entirely driven by increased drive densities: http://queue.acm.org/detail.cfm?id=1670144

The reality is the higher drive densities mean you lose more bytes at a time when you have a drive failure, and that means more bytes you want to have "recovered".

Re: FreeNAS: Open Source Storage Operating System

#174

Post by baruch »

cbsmith wrote: post #172
baruch wrote: post #58

Earlier quoted context omitted.

An HDD/SSD without ECC will not work, and wouldn't have worked for a long time now. Many of the reads performed will require some ECC use. ECC is always written alongside the actual block and the overhead for ECC is the reason for the move from 512b sectors to 4Kb sectors in HDDs. For SSDs the data is already written in different block sizes depending on the NAND and the internal representation and ECC is done for la…

> An HDD/SSD without ECC will not work, and wouldn't have worked for a long time now. Many of the reads performed will require some ECC use. Yeah, I expressed that badly. I always forget about the ECC bytes that are in the firmware. > The probability of failure during rebuild is not really directly linked to drive size, the usual interpretation of the drive BER is wrong (media BER is stated across a large population…

Adam Leventhal uses the wrong interpretation of the BER value so I wouldn't take his words at face value. The whole argument of Adam is by the disk BER and not by density.

I do assume though that the RAID array does media scrub (BMS) periodically, if you don't you are at risk anyway and I'd call that negligence in maintaining your RAID.

If you do use scrubbing the risk that another drive has a bad media spot is low as it must have developed in the time from the last scan and that is a bounded time (week, two, four) so the risk of two drives having a bad sector is now even lower (though never non-zero, backups are still a thing). If you couple that also with TLER and proper media handling instead of dropping the disk on media error the risk to the data becomes very low since there isn't a very high likelyhood that two disks will have a bad sector in the same stripe.

I've been working with HDDs and SSDs and developing software for enterprise storage systems for a number of years now, I've worked in XIV and for all the thousands of systems and hundred of thousand disks of many models I've never seen two disks fail to read the same stripe and RAID recovery was always possible. Other problems are more likely sooner than an actual RAID failure (technician shutting down the system by pressing the UPS buttons or a software bug).

I did learn of several failure modes that can increase the risk but they depend on specific workloads that are not generally applicable. One of those is that if you write to one track all too often you may affect nearby tracks and if the workload is high enough you don't give the disk the time to fix this (the disk tracks this failure mode and will workaround it in idle time). In such a case same stripe can be affected in multiple disks and the time to develop may (or may not) be shorter than the media scrub time. And even then a thin provisioned raid structure would reduce the risk of this failure mode and giving disks some rest time (even just a few seconds) would allow the drive to fix this and other problems it knows about.

All in all, RAID is not dead (yet).

Re: FreeNAS: Open Source Storage Operating System

#175

Post by Annatar »

mindslight wrote: post #170
Annatar wrote: post #158

Earlier quoted context omitted.

What is "unison"?

Bidirectional file synchronization. It makes any single replica not that important. Don't concentrate your resources into a single "invincible" box.

Is the synchronization synchronous or asynchronous?

Is there a uniform resource identifier where I could read more about it? Or perhaps some book? Research paper?

Re: FreeNAS: Open Source Storage Operating System

#176

Post by mindslight »

Annatar wrote: post #175
mindslight wrote: post #170

Earlier quoted context omitted.

Bidirectional file synchronization. It makes any single replica not that important. Don't concentrate your resources into a single "invincible" box.

Is the synchronization synchronous or asynchronous? Is there a uniform resource identifier where I could read more about it? Or perhaps some book? Research paper?

https://www.cis.upenn.edu/~bcpierce/unison/

It synchronizes directory trees on arbitrary filesystems, so I think the answer to your question would be "asynchronous".

Most individuals don't have TBs of working set continually being updated and requiring a central authoritative copy. They have a small working set and a long-term archive they'd just never want to lose.

Unison's model isn't perfect (eg having to create a star topology, lack of built-in inotify). But it's been around forever, is written in a sane language, and is rock solid.

Re: FreeNAS: Open Source Storage Operating System

#177

Post by tracker1 »

ta_donk_gt wrote: post #48
tracker1 wrote: post #8

Although I don't think I'd ever DIY another NAS box, I would suggest the following, use a common chipset controller that exposes each drive/device separately with all information, some don't. Also, prefer a system with ECC, Asus supports ECC on most AMD FX supporting motherboards for the less expensive option there. Also, have 2 spare drives on hand. I went back to Synology after my uper-nas crashed and burned (reall…

I had a similar bad run with Seagate 3TB drives..3 out of 4 in 2 years. I was really thinking it was the NAS, but switched to WD Reds and have had no issues in 18 months.

That isn't to say that reds/greens don't have issues... I've seen one of each fail, but they have been very solid compared to my experience with Seagate drives... the shame is that may very well be different today, but brand (dis)loyalty tends to carry for longer terms.

Re: FreeNAS: Open Source Storage Operating System

#178

Post by rleigh »

voltagex_ wrote: post #162
iberinger wrote: post #104

Earlier quoted context omitted.

Agreed, mirror vdevs are the way to go. Here's a post that explains the tradeoffs between RAIDZ and mirror vdevs in great detail: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...

I'm still not sure what setup I have. I really should have written down every decision I've made while building my NAS (I think I've now found articles that disagree with every single choice I made) $ zpool status pool: BoxODisks state: ONLINE scan: scrub repaired 0 in 9h20m with 0 errors on Sun Jul 31 01:31:31 2016 config: NAME STATE READ WRITE CKSUM BoxODisks ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/18059e22-b6a4-1…

When you want to upgrade, you can offline and replace each disk, then resilver, in turn. If the pool is set to autoexpand, you'll eventually get a total capacity of 16TB. You can do this for all the disks, or for each mirror separately in two groups of two.

Re: FreeNAS: Open Source Storage Operating System

#179

Post by bgmccollum »

jdhawk wrote: post #55
ibelimb wrote: post #7

Earlier quoted context omitted.

Supposedly Xpenology[0] lets you do that. I have not used it personally, as I have actual Synology hardware. [0] http://xpenology.org/

Yes, on x86 systems it mimics a DS3615. I've seen it installed on cheap dell desktops, like an optiplex 755. Works well, with some limitations. I dont think their automatic port forwarding service (quickconnect) works well - since that service checks for a valid Synology MAC address. Additionally - you can't do the automatic system updates as it might break certain portions that XPEnology overrides.

There is a serial number / MAC generator that makes quick connect work just fine.

Re: FreeNAS: Open Source Storage Operating System

#180

Post by cbsmith »

baruch wrote: post #174
cbsmith wrote: post #172

Earlier quoted context omitted.

> An HDD/SSD without ECC will not work, and wouldn't have worked for a long time now. Many of the reads performed will require some ECC use. Yeah, I expressed that badly. I always forget about the ECC bytes that are in the firmware. > The probability of failure during rebuild is not really directly linked to drive size, the usual interpretation of the drive BER is wrong (media BER is stated across a large population…

Adam Leventhal uses the wrong interpretation of the BER value so I wouldn't take his words at face value. The whole argument of Adam is by the disk BER and not by density. I do assume though that the RAID array does media scrub (BMS) periodically, if you don't you are at risk anyway and I'd call that negligence in maintaining your RAID. If you do use scrubbing the risk that another drive has a bad media spot is low a…

So why would one ever bother with RAID-6 (or its moral equivalent with RAID-Z)? I've yet to hear someone justify it from a requirement being to actually survive two drives failing simultaneously.
Post reply on HN