Earlier quoted context omitted.
Sounds great in theory, but in practice you'll be having that conversation about your database schema anyway.
Nah, just throw it all into a NoSQL store and let the developers figure it out.
Is that a joke?
Switch Your Databases To Flash Storage
51–60 of 80 posts
Re: Switch Your Databases To Flash Storage
#52Earlier quoted context omitted.
Note he didn't say 10% premium for the same capacity. It's possible for cloud providers to replace $50 hard disks with $64 SSDs today.
Does a $64 SSD have enough charge to flush pending writes after a power failure? If not, then be prepared for widespread corruption. See comment by pjungwir.
You're already on a cloud system, so you have to have a plan in place for your instance to up and disappear without warning. If your instance has an unplanned outage of any kind, you kill it and spawn a new one. You may as well use libeatmydata and reap the performance benefit.
Re: Switch Your Databases To Flash Storage
#53Earlier quoted context omitted.
Does a $64 SSD have enough charge to flush pending writes after a power failure? If not, then be prepared for widespread corruption. See comment by pjungwir.
You're already on a cloud system, so you have to have a plan in place for your instance to up and disappear without warning. If your instance has an unplanned outage of any kind, you kill it and spawn a new one. You may as well use libeatmydata and reap the performance benefit.
You are describing EC2. Most other providers do provide proper persistence.
Re: Switch Your Databases To Flash Storage
#54Wear patterns and flash are an issue, although rotational drives fail too. There are several answers. When a flash drive fails, you can still read the data. A clustered database and multiple copies of the data, you gain reliability – a server level of RAID. As drives fail, you replace them. Unlike magnetic disks, SSDs have a tendency to fail at a really predictable rate. So predictably that if you've got two drives o…
Just curious, which software do you recommend to monitor SSDs life span/upcoming death in production?
Can something like Munin do this out of the box?-
Re: Switch Your Databases To Flash Storage
#55Re: Switch Your Databases To Flash Storage
#56Wear patterns and flash are an issue, although rotational drives fail too. There are several answers. When a flash drive fails, you can still read the data. A clustered database and multiple copies of the data, you gain reliability – a server level of RAID. As drives fail, you replace them. Unlike magnetic disks, SSDs have a tendency to fail at a really predictable rate. So predictably that if you've got two drives o…
> That said, if you're careful then that predictability should be a good thing. Yes, it's a very good thing. In a high end SSD storage system, you predict early enough based on a calculation of how many drives there are, and what their current wear is, what type they are (SLC, eMLC, cMLC), etc. Then you phone home and have a drive delivered before the user even sees a disk failure. With HDD's, the failure rate is so…
It's hard to predict without SMART features to measure the current wear state. Write amplification from the file system, and from the drive itself if you're not using large block writes, means you can't just calculate - you have to measure.
Re: Switch Your Databases To Flash Storage
#57Anyone have any idea when Amazon will start providing SSD-backed RDS?
They already do, although it is quite new. The AWS provisioned IOPS layer ("pIOPS") is SSD-backed, and can be used for RDS:
http://aws.amazon.com/rds/#PIOPS
There isn't currently an easy way to migrate to pIOPS from traditional RDS, but the performance is fantastic and works as advertised.
Re: Switch Your Databases To Flash Storage
#58The PostgreSQL mailing list is having a conversation right now about using SSDs. This seems like a very important comment for anyone considering them: http://archives.postgresql.org/pgsql-general/2012-12/msg00202.php Basically, you need to make sure that you buy SSDs with a capacitor that allows the drive to flush what it needs in event of abrupt power loss. EDIT: Looks like the list archives didn't preserve the thre…
Is this because PostgreSQL's file format will become hopelessly confused and unable to restart if flush doesn't work?
A lot of applications can lose the last 100ms of writes, especially if its rare because of a k-safe cluster design, as long as you don't have a corrupted file format. A good transaction log based system will recover - as the author's should.
Re: Switch Your Databases To Flash Storage
#59Wear patterns and flash are an issue, although rotational drives fail too. There are several answers. When a flash drive fails, you can still read the data. A clustered database and multiple copies of the data, you gain reliability – a server level of RAID. As drives fail, you replace them. Unlike magnetic disks, SSDs have a tendency to fail at a really predictable rate. So predictably that if you've got two drives o…
Just curious, which software do you recommend to monitor SSDs life span/upcoming death in production? Can something like Munin do this out of the box?-
Munin cannot predict a SSD's lifespan.
Re: Switch Your Databases To Flash Storage
#60I don't disagree with the conclusions, but don't you have to short stroke those ssds pretty significantly in a high transaction environment to avoid write amplification? It's too bad longevity worries are keeping them out of the no commitment market.
Not really. "short stroking" is called "overprovisioning" with SSDs, and you'll see different effects with different drives. The magic number with most consumer SSDs (the mentioned Intel and Samsung drives) do best with about 20% overprovisioning. The "enterprise class" drives don't require this - they bake in the overprovisioning. The new Intel s3700 works extraordinarily well with no overprovisioning.