Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When Solid State Drives Are Not That Solid

blog.algolia.com

Post reply on HN
Print view View original

51–60 of 123 posts

Re: When Solid State Drives Are Not That Solid

#51

Post by jhead »

adamsurak wrote: post #46
jlebar wrote: post #36

To me, this sort of thing brings home the value of not running your own machines. Sure, Amazon's/Google's clouds have quirks, but it's far less likely that you're going to have to debug faulty hardware in this way. It sounds like a team of more than one person worked on this at least part-time for weeks -- how much is that worth? It's not just the cost of hiring extra people to do the work; often small companies simp…

With our architecture, HW requirements, the price of HW and the price of the cloud VMs, even working on this for a week or two saves us significant amount of money both short-term and long-term. The side effect is that we now have tools to recover servers way faster and allows us to do things we have not thought about before.

Agreed. Additionally, some business models simply don't mesh with cloud infrastructure pricing no matter the volume. There are definitely advantages to using cloud services, but most of the time bare metal gets you more hardware/performance at a lower cost in the long run, even when you factor in everything else that it entails.

Re: When Solid State Drives Are Not That Solid

#52

Post by adamsurak »

cft wrote: post #28

Using SAS SSD drives on a server is a bad idea for many reasons. One should use PCIe cards, that sit directly on the PCIe bus, such as FusionIO or SanDisk. They have been tested and retested (e.g. by Facebook), without the unnecessarily added complexity of SAS/SATA protocols. The I/O performance is also about 20x.

I completely agree and we are going this direction.

Re: When Solid State Drives Are Not That Solid

#53

Post by ploxiln »

adamsurak wrote: post #45
ploxiln wrote: post #3

Originally TRIM was an un-queued command; all writes had to be flushed, then TRIM executed, then writes could continue. This was bad for performance with automatic on-file-delete trim, so everyone wanted a trim command that could be put in the command queue along with writes. Many new drives have this. It turns out that Samsung 8XX SSDs advertise they support queued trim but it's buggy. The old TRIM command works fin…

In this case the TRIM command was un-queued, which makes it worse.

It sounds to me like even when it's the fstrim utility, which uses some ioctl() to tell the kernel to trim free regions in a range on a filesystem, the kernel ends up causing the queued trim command to be used if available.

The "blacklist" does not appear to have any constant to blacklist old-style trim, only NCQ_TRIM (and other odd stuff, most notably all NCQ usage).

This makes sense, because if some SSD advertised old-style trim but was corrupted by it, then it would be found and fixed sooner by these vendors, because Windows 7 would exhibit the corruption.

Re: When Solid State Drives Are Not That Solid

#54

Post by ptaipale »

jjawssd wrote: post #50
_yosefk wrote: post #43

Earlier quoted context omitted.

The thing is, almost all hardware accessed through drivers has tons of bugs, at least it's nowhere near as close to "bug-free" as are things like CPUs or DRAMs which cannot hide their bugs behind drivers. The thing that one can hope to work reasonably is a piece of hardware plus an accompanying driver which knows to hide that hardware's issues. So another way of putting what you said would be "on Linux there's no wor…

If you think CPUs do not come with a shit-ton of hardware bugs YOU ARE GRAVELY MISTAKEN. Google up the Intel errata for the i7 The list goes on and on.

I didn't see him thinking that. Just that CPUs do not have as many bugs as other hardware - which I think is quite true. With CPUs a larger portion of bugs are found, and smaller bugs matter because they are not hidden by proprietary drivers.

Re: When Solid State Drives Are Not That Solid

#55

Post by MichaelCrawford »

ploxiln wrote: post #3

Originally TRIM was an un-queued command; all writes had to be flushed, then TRIM executed, then writes could continue. This was bad for performance with automatic on-file-delete trim, so everyone wanted a trim command that could be put in the command queue along with writes. Many new drives have this. It turns out that Samsung 8XX SSDs advertise they support queued trim but it's buggy. The old TRIM command works fin…

"workarounds in the kernel."

Please permit me to violate my NDA:

/* MacWrite needs this */

... in Mac OS System 7.5.2. I honestly don't know whether MacWrite still needed it but that code was there to work around a bug.

Re: When Solid State Drives Are Not That Solid

#56

Post by MichaelCrawford »

digi_owl wrote: post #24
ploxiln wrote: post #3

Originally TRIM was an un-queued command; all writes had to be flushed, then TRIM executed, then writes could continue. This was bad for performance with automatic on-file-delete trim, so everyone wanted a trim command that could be put in the command queue along with writes. Many new drives have this. It turns out that Samsung 8XX SSDs advertise they support queued trim but it's buggy. The old TRIM command works fin…

In essence the Linux kernel put on display what is on Windows hidden by proprietary device drivers.

that's one of the things drivers are for; to workaround hardware bugs.

Among the challenges faced by the AMCC 3ware RAID HBAs were faulty motherboards.

"But PCI is a standard!" you quite reasonably protest.

Yes, and the US Constitution guarantees us many inalienable rights.

Re: When Solid State Drives Are Not That Solid

#57

Post by rleigh »

ThatPlayer wrote: post #8
douglasheriot wrote: post #5

Wow, that sucks. Another reason to use ZFS – you’d notice the corrupted files a lot sooner.

Or Btrfs on Linux.

In theory, yes. Unfortunately, every time my Btrfs filesystems have encountered a hardware glitch, it has happily trashed the filesystem beyond recovery (including both drives in a RAID1 mirror, one of which was perfectly OK). I use ZFS now, and while some features are compatable with Btrfs, the implementation quality, documentation, and feature completeness, and tool quality set it well above where Btrfs is at.

Re: When Solid State Drives Are Not That Solid

#58

Post by hvidgaard »

jhead wrote: post #51
adamsurak wrote: post #46

Earlier quoted context omitted.

With our architecture, HW requirements, the price of HW and the price of the cloud VMs, even working on this for a week or two saves us significant amount of money both short-term and long-term. The side effect is that we now have tools to recover servers way faster and allows us to do things we have not thought about before.

Agreed. Additionally, some business models simply don't mesh with cloud infrastructure pricing no matter the volume. There are definitely advantages to using cloud services, but most of the time bare metal gets you more hardware/performance at a lower cost in the long run, even when you factor in everything else that it entails.

The thing people forgets, is that the cloud provider have the same issues and expences. That cost is passed on to the clients. Now they may be more efficient ect. but once you reach a certain scale, and it's less that people think, you might as well get it done in house if you can find qualified people.

Re: When Solid State Drives Are Not That Solid

#59

Post by digi_owl »

jjawssd wrote: post #50
_yosefk wrote: post #43

Earlier quoted context omitted.

The thing is, almost all hardware accessed through drivers has tons of bugs, at least it's nowhere near as close to "bug-free" as are things like CPUs or DRAMs which cannot hide their bugs behind drivers. The thing that one can hope to work reasonably is a piece of hardware plus an accompanying driver which knows to hide that hardware's issues. So another way of putting what you said would be "on Linux there's no wor…

If you think CPUs do not come with a shit-ton of hardware bugs YOU ARE GRAVELY MISTAKEN. Google up the Intel errata for the i7 The list goes on and on.

Heh, i still recall my early encounters with Linux and reading the bootup messages.

One of them contained a line related to having found a CPU bug and having put a workaround in place.

I am not entirely sure, but i think it may have been the F00F bug.

https://en.wikipedia.org/wiki/Pentium_F00F_bug

Re: When Solid State Drives Are Not That Solid

#60

Post by adamsurak »

kbar13 wrote: post #7

if one machine failed and failover kicked in correctly, why was the engineer paged?

We have multiple different pages. In our cluster we have 3 machines and if one of them is unavailable because of broken network, we do not page. In this case the page came as an application error that the application was not able to cope with. When we have issue that we have seen before and the server can handle it on its own, we do not page.
Post reply on HN