Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Switch Your Databases To Flash Storage

highscalability.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Switch Your Databases To Flash Storage

#71

Post by jacques_chester »

buro9 wrote: post #12

I would love if cloud providers offered SSD options for their full range of boxes. For example, to be able to get a Linode at only a fraction more of the cost (say, a 10% premium) with the disk being SSD (and obviously reduced capacity compared to HDD). I have seen the current offerings but found them to either be too costly (AWS, only one of the the largest instances), or too onerous (ssdnodes.com whose base product…

In Australia, OrionVM offer fully SSD-backed VMs.

It's cost-effective because in Australia, the major determinant of monthly hosting bills is our ludicrously overpriced bandwidth fees.

Re: Switch Your Databases To Flash Storage

#72

Post by baruch »

ChuckMcM wrote: post #40
baruch wrote: post #30

Earlier quoted context omitted.

Assuming this predictability is not a good idea in my experience. SSDs fail in various ways, some may be predictable and some are completely unpredictable. It is also not true that an ssd failure means it simply goes to readonly mode. I've seen plenty of SSDs failing unexpectedly and are no longer readable, returning sense key 0x4 (HARDWARE ERROR) and the only recourse is to ship them out. The risk of correlated fail…

We've currently got about 3500 SSDs in production across our clusters. I worry about them deciding to all fail at once, so far they have been sporadic failures (about 1/2 of which leave the drive unusable).

Are they all the same model? How long have they been running? Is it a relatively similar load on all of them? Can you share smart attributes for them? (in private if needed)

On a very low fire I'm trying to create a disk survey project (http://disksurvey.org ) and such information is of great interest to me.

Re: Switch Your Databases To Flash Storage

#73

Post by baruch »

pjungwir wrote: post #69
dclusin wrote: post #63

Earlier quoted context omitted.

Disk drives with battery backed write caches are a requirement for any system where data corruption is considered a system failure. We use these at my job for things other than just our data bases. It doesn't need to be just SSD's either; traditional platter-based hard drives have this feature as well.

Thanks; I wondered why this would be unique to SSDs.

Normally you use an HDD without any write cache and if you do use a write cache along the way (on the HDD or on the server) you make sure you are battery backed.

SSD is a bit different since even if you do not use write caching on the SSD there is a non-trivial amount of meta data that is kept in RAM and needs to be written safely to the media. You need quite a bit of juice to do all this work which is entails keeping most of the hardware working.

Even the HDD has enough capacitor power to park the head back and lock it safely. But I believe that you don't really need all hardware operating at full capacity, only enough spin to rotate and to pull the head back and the spin continues even if you don't power it for the parking time.

Both HDD and SSD do not guarantee much about an IO that was started writing but an acknowledgement wasn't sent about it. SCSI standard from which all disks derive requirements doesn't require anything in such a case and leaves it as undefined.

Re: Switch Your Databases To Flash Storage

#74

Post by codemac »

bbulkow wrote: post #56
codemac wrote: post #41

Earlier quoted context omitted.

> That said, if you're careful then that predictability should be a good thing. Yes, it's a very good thing. In a high end SSD storage system, you predict early enough based on a calculation of how many drives there are, and what their current wear is, what type they are (SLC, eMLC, cMLC), etc. Then you phone home and have a drive delivered before the user even sees a disk failure. With HDD's, the failure rate is so…

It's hard to predict without SMART features to measure the current wear state. Write amplification from the file system, and from the drive itself if you're not using large block writes, means you can't just calculate - you have to measure.

I'm sorry, I didn't mean you'd calculate some value once for all the drives. It's definitely something you measure over the lifespan of the SSD itself with the rest of your QoS subsystem.

Reads, program/erases, controller ecc/ read disturb management, the g/p list mapping of the blocks... This all has to be taken into account in a dynamic way. And yes, some people are doing this at a higher level than the SSD controller.

Re: Switch Your Databases To Flash Storage

#75

Post by codemac »

moe wrote: post #67
bbulkow wrote: post #56

Earlier quoted context omitted.

It's hard to predict without SMART features to measure the current wear state. Write amplification from the file system, and from the drive itself if you're not using large block writes, means you can't just calculate - you have to measure.

To extend on your comment: It's not possible to predict individual drive failure with reasonable accuracy. It's disconcerting to see the parent comment suggesting this still sits at the top of the thread. You can roughly predict the longest possible lifespan for a SSD under a given workload. Regardless, a significant percentage of drives will still die earlier than that.

I'm only saying this from experience developing storage systems that are yet unreleased. You can predict the lifespan of the SSD in the storage system if you give up many of the functions of the SSD controller and put them in software RAID.

If you're talking about most incredibly naive SSD storage systems available today (excluding violin memory and maybe xtreme/pure), then I agree with you.

Re: Switch Your Databases To Flash Storage

#76

Post by dutchbrit »

Funny, and it's a no brainer really.. There was a thread about SSD's about 2 years back, regarding good ways to use them. My conclusion was pretty much the same when it came to DB's, yet nobody agreed with me back then and I received 3 downvotes. Odd!!

Good article!

Re: Switch Your Databases To Flash Storage

#77

Post by lasonrisa »

staunch wrote: post #11

Shameless plug alert. At Uptano[1], this is one of the neatest things we've seen with our very inexpensive SSD machines. It's amazing what you can do with 8GB RAM + 100 GB RAID1 SSD. It's probably the best price:performance DB you can run, and is sufficient for ~95% of projects. 1. https://uptano.com

Hey, nice relooking of your site. I prefer this colour palette better.

Re: Switch Your Databases To Flash Storage

#78

Post by hvidgaard »

moe wrote: post #61
bunderbunder wrote: post #14

Wear patterns and flash are an issue, although rotational drives fail too. There are several answers. When a flash drive fails, you can still read the data. A clustered database and multiple copies of the data, you gain reliability – a server level of RAID. As drives fail, you replace them. Unlike magnetic disks, SSDs have a tendency to fail at a really predictable rate. So predictably that if you've got two drives o…

So all you've got to do is monitor the status of the drives, and replace them before they get too close to their rated lifespan Sorry, that is terrible advice. Do not do that. In availability planning two is one and one is zero. If you love your data you run your databases in pairs. If you really love your data you run them in triplets. This applies no matter what disk technology you're using. Speculations about fail…

Not only that - you make sure that critical components, like hard drives (rotational or SSD, doesn't matter), are from different manufacturers or at least not the same production, or at the very least not put them to use at the same time. Basic design flaws (intentional or not), that result in non functional hardware, tend to hit at the same time - so you'd rather not want to have all 12 drives, and the hard drives of the 3 replicas, fail within the same week.

Re: Switch Your Databases To Flash Storage

#79

Post by bunderbunder »

moe wrote: post #61
bunderbunder wrote: post #14

Wear patterns and flash are an issue, although rotational drives fail too. There are several answers. When a flash drive fails, you can still read the data. A clustered database and multiple copies of the data, you gain reliability – a server level of RAID. As drives fail, you replace them. Unlike magnetic disks, SSDs have a tendency to fail at a really predictable rate. So predictably that if you've got two drives o…

So all you've got to do is monitor the status of the drives, and replace them before they get too close to their rated lifespan Sorry, that is terrible advice. Do not do that. In availability planning two is one and one is zero. If you love your data you run your databases in pairs. If you really love your data you run them in triplets. This applies no matter what disk technology you're using. Speculations about fail…

You're responding as if I had suggested that this is a replacement for all the other practices one should already be doing.

If I had, yes I would agree with you 100%. However, far from suggesting anything remotely like that, I made sure to work in the phrase "add that extra step." It's not a panacea, it's an additional thing that needs to be done to account for one new quirk that a particular technology throws into the mix.

Re: Switch Your Databases To Flash Storage

#80

Post by vidarh »

hvidgaard wrote: post #78
moe wrote: post #61

Earlier quoted context omitted.

So all you've got to do is monitor the status of the drives, and replace them before they get too close to their rated lifespan Sorry, that is terrible advice. Do not do that. In availability planning two is one and one is zero. If you love your data you run your databases in pairs. If you really love your data you run them in triplets. This applies no matter what disk technology you're using. Speculations about fail…

Not only that - you make sure that critical components, like hard drives (rotational or SSD, doesn't matter), are from different manufacturers or at least not the same production, or at the very least not put them to use at the same time. Basic design flaws (intentional or not), that result in non functional hardware, tend to hit at the same time - so you'd rather not want to have all 12 drives, and the hard drives o…

I used to run a free webmail service back in '99-2000. It was the first system of that kind of scale I'd worked on (about 1.5 million user accounts - large by the standards of the day - today I have more storage in my home fileserver; heck, I've got almost as much storage in my laptop), and though I wasn't in charge of ordering hardware I was equally oblivious to this problem as the guy who did.

I learned in a way that ensures this advice is burned into my memory forever:

It was when IBM had one of their worst ever manufacturing problems for one of their drive ranges.

While the IBM distributor we dealt with was very fast at turning around replacement drives, we had some nerve-wrecking weeks when the second drive in one of our arrays failed only something like 6-9 months after we went live, and we found out about the problem.

They all failed one after the other within a week or two of each other. Every drive in our main user mailbox storage array...

Thankfully for us, the gap was long enough between each failure that the array was rebuilt and then some in between each failure, but we spent a disproportionate amount of time babysitting backups and working on contingency plans because we'd made that stupid mistake.

(And I'll never ever build out a single large array, or any number of other thing - it made me spend a lot of time thinking about and reading up on redundancy and disaster recovery strategies, as it scared the hell out of me; it was mostly luck that prevented us from losing a substantial amount of data)

Post reply on HN