Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Switch Your Databases To Flash Storage

highscalability.com

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Switch Your Databases To Flash Storage

#61

Post by moe »

bunderbunder wrote: post #14

Wear patterns and flash are an issue, although rotational drives fail too. There are several answers. When a flash drive fails, you can still read the data. A clustered database and multiple copies of the data, you gain reliability – a server level of RAID. As drives fail, you replace them. Unlike magnetic disks, SSDs have a tendency to fail at a really predictable rate. So predictably that if you've got two drives o…

So all you've got to do is monitor the status of the drives, and replace them before they get too close to their rated lifespan

Sorry, that is terrible advice. Do not do that.

In availability planning two is one and one is zero.

If you love your data you run your databases in pairs. If you really love your data you run them in triplets. This applies no matter what disk technology you're using.

Speculations about failure rate or -prediction don't belong here. Your server can go up in flames at any time for a dozen reasons, the disks being only one of them.

Re: Switch Your Databases To Flash Storage

#62

Post by hosay123 »

bbulkow wrote: post #58
pjungwir wrote: post #39

The PostgreSQL mailing list is having a conversation right now about using SSDs. This seems like a very important comment for anyone considering them: http://archives.postgresql.org/pgsql-general/2012-12/msg00202.php Basically, you need to make sure that you buy SSDs with a capacitor that allows the drive to flush what it needs in event of abrupt power loss. EDIT: Looks like the list archives didn't preserve the thre…

Is this because PostgreSQL's file format will become hopelessly confused and unable to restart if flush doesn't work? A lot of applications can lose the last 100ms of writes, especially if its rare because of a k-safe cluster design, as long as you don't have a corrupted file format. A good transaction log based system will recover - as the author's should.

It's not only a case of file formats.. a few years ago one of the Linux kernel developers (Theodore Tso IIRC) made a post regarding drive behaviour under power loss, and the results were pretty insane.

For example when a rotating drive fails, you might lose +/- 4kb around the previous sector under write, whereas with particular SSDs, he witnessed 1mb chunks zeroed out every Nmb across the entire drive. That kind of thing, you simply can't work around in software

Re: Switch Your Databases To Flash Storage

#63

Post by dclusin »

pjungwir wrote: post #39

The PostgreSQL mailing list is having a conversation right now about using SSDs. This seems like a very important comment for anyone considering them: http://archives.postgresql.org/pgsql-general/2012-12/msg00202.php Basically, you need to make sure that you buy SSDs with a capacitor that allows the drive to flush what it needs in event of abrupt power loss. EDIT: Looks like the list archives didn't preserve the thre…

Disk drives with battery backed write caches are a requirement for any system where data corruption is considered a system failure. We use these at my job for things other than just our data bases. It doesn't need to be just SSD's either; traditional platter-based hard drives have this feature as well.

Re: Switch Your Databases To Flash Storage

#64

Post by knappador »

Those caught in the middle on DB size needs and performance would be well off to take a look at Bcache. http://bcache.evilpiepirate.org/ It's a block write-back cache and seems to perform really nicely. Here's some benchmarks. http://www.accelcloud.com/2012/04/18/linux-flashcache-and-bc...

Re: Switch Your Databases To Flash Storage

#65

Post by jQueryIsAwesome »

venomsnake wrote: post #20
bunderbunder wrote: post #14

Wear patterns and flash are an issue, although rotational drives fail too. There are several answers. When a flash drive fails, you can still read the data. A clustered database and multiple copies of the data, you gain reliability – a server level of RAID. As drives fail, you replace them. Unlike magnetic disks, SSDs have a tendency to fail at a really predictable rate. So predictably that if you've got two drives o…

But with current SSD speed/size ratio this vulnerability window can be only a few minutes, also this can be minimized with mixing batches and vendors of drives.

Or just mixing intervals of memory installation; so you install the next SSD at half the life-span of the previous one.

Re: Switch Your Databases To Flash Storage

#67

Post by moe »

bbulkow wrote: post #56
codemac wrote: post #41

Earlier quoted context omitted.

> That said, if you're careful then that predictability should be a good thing. Yes, it's a very good thing. In a high end SSD storage system, you predict early enough based on a calculation of how many drives there are, and what their current wear is, what type they are (SLC, eMLC, cMLC), etc. Then you phone home and have a drive delivered before the user even sees a disk failure. With HDD's, the failure rate is so…

It's hard to predict without SMART features to measure the current wear state. Write amplification from the file system, and from the drive itself if you're not using large block writes, means you can't just calculate - you have to measure.

To extend on your comment: It's not possible to predict individual drive failure with reasonable accuracy. It's disconcerting to see the parent comment suggesting this still sits at the top of the thread.

You can roughly predict the longest possible lifespan for a SSD under a given workload. Regardless, a significant percentage of drives will still die earlier than that.

Re: Switch Your Databases To Flash Storage

#68

Post by peschkaj »

bbulkow wrote: post #56
codemac wrote: post #41

Earlier quoted context omitted.

> That said, if you're careful then that predictability should be a good thing. Yes, it's a very good thing. In a high end SSD storage system, you predict early enough based on a calculation of how many drives there are, and what their current wear is, what type they are (SLC, eMLC, cMLC), etc. Then you phone home and have a drive delivered before the user even sees a disk failure. With HDD's, the failure rate is so…

It's hard to predict without SMART features to measure the current wear state. Write amplification from the file system, and from the drive itself if you're not using large block writes, means you can't just calculate - you have to measure.

Depending on the SSD vendor, many drives expose performance counters to help you estimate wear level.

Re: Switch Your Databases To Flash Storage

#69

Post by pjungwir »

dclusin wrote: post #63
pjungwir wrote: post #39

The PostgreSQL mailing list is having a conversation right now about using SSDs. This seems like a very important comment for anyone considering them: http://archives.postgresql.org/pgsql-general/2012-12/msg00202.php Basically, you need to make sure that you buy SSDs with a capacitor that allows the drive to flush what it needs in event of abrupt power loss. EDIT: Looks like the list archives didn't preserve the thre…

Disk drives with battery backed write caches are a requirement for any system where data corruption is considered a system failure. We use these at my job for things other than just our data bases. It doesn't need to be just SSD's either; traditional platter-based hard drives have this feature as well.

Thanks; I wondered why this would be unique to SSDs.

Re: Switch Your Databases To Flash Storage

#70

Post by jacques_chester »

bbulkow wrote: post #58
pjungwir wrote: post #39

The PostgreSQL mailing list is having a conversation right now about using SSDs. This seems like a very important comment for anyone considering them: http://archives.postgresql.org/pgsql-general/2012-12/msg00202.php Basically, you need to make sure that you buy SSDs with a capacitor that allows the drive to flush what it needs in event of abrupt power loss. EDIT: Looks like the list archives didn't preserve the thre…

Is this because PostgreSQL's file format will become hopelessly confused and unable to restart if flush doesn't work? A lot of applications can lose the last 100ms of writes, especially if its rare because of a k-safe cluster design, as long as you don't have a corrupted file format. A good transaction log based system will recover - as the author's should.

> A good transaction log based system will recover - as the author's should.

Unless the drive lies. Tells filthy dirty ugly lies.

And lots of them do.

OS: "Did you persist that data I flushed?"

Disk controller (hiding single copy in RAM cache behind its back): "Oh absolutely, bro. It's tight. Solid. You got nothing to worr-"BZZZT

Post reply on HN