Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Server Status

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 124 posts

Re: Tell HN: Server Status

#111

Post by chrissnell »

JoeAltmaier wrote: post #39
barrkel wrote: post #35

By tolerating the loss of two disks, do you mean raidz2 or do you mean 3-way mirror? Raidz2 is not fast. In fact, it is slow. Also, it is less reliable than a two way mirror in most configurations, because recovering from a disk loss requires reading the entirety of every other disk, whereas recovering from loss in a mirror requires reading the entirety of one disk. The multiplication of the probabilities don't work…

Reading 2 disks is not very much slower than 1. Its always the I/o bandwidth that is at issue, so it very much depends on the connectivity. And it all pretty much impacts your customer. We've all suffered under the 4-hour mirror rebuild, the whole machine made inoperable by the constant disk load. The only way to alleviate that, is to design extra bandwidth (e.g. another cable and controller) that's used exclusively…

Can you elaborate on the second controller? Are you referring to two disks on two controllers, using mdadm (or similar) to create a software RAID 1 across the two?

Re: Tell HN: Server Status

#112

Post by stock_toaster »

whalesalad wrote: post #106

There's a lot of tuning that can be done on a ZFS setup to improve performance. I'm not a pro, so others will have more feedback and knowledge, but some things off the top of my head to get you started: Add a flash memory based (SSD) ZIL or L2ARC or both to the box. That'll help improve read/write performance. I believe the ZIL (ZFS intent log) is used to cache during writes, and the L2ARC is used during reads. You m…

> I believe the ZIL (ZFS intent log) is used to cache during writes, and the L2ARC is used during reads.

I think the ZIL (zfs intent log) is an intermediary for synchronous writes only. My understanding is that it effectively turns the sync write into an async write (from the standpoint of the zpool) -- this is why it requires a faster device than the pool it is used with. If it is absent, the pool itself houses the zil.

Re: Tell HN: Server Status

#113

Post by takeda »

whalesalad wrote: post #106

There's a lot of tuning that can be done on a ZFS setup to improve performance. I'm not a pro, so others will have more feedback and knowledge, but some things off the top of my head to get you started: Add a flash memory based (SSD) ZIL or L2ARC or both to the box. That'll help improve read/write performance. I believe the ZIL (ZFS intent log) is used to cache during writes, and the L2ARC is used during reads. You m…

ARC basically is a read cache.

ZIL improves performance of writes, but by itself is almost never read (only written to) except on failures. It will be read for example on power failure to finish writing the data to the disk. It is used to speed up synchronous writes.

Re: Tell HN: Server Status

#114

Post by ivoras »

scurvy wrote: post #101

Why on earth are you not using SSD's? The HN footprint can't be that large. The extra speed and reliability from a pair of SSD's has to far outweigh the costs.

I'd guesstimate that the READ load is served practically entirely from RAM (file cache) and the WRITE load is non-critical enough that it's done "eventually consistent" (e.g. synchronous_commit=off in PostgreSQL, or fsync=off elsewhere) - or at least that's how I'd run it. YMMV.

Re: Tell HN: Server Status

#115

Post by lsc »

Are you bottlenecking on high iowait? or something else?

just one random bit to try... Obviously, I have no insight into your system and I'm not saying I know more than you or anything, but I've been seeing more situations lately where I had massive latency but reasonable throughput and the disks mostly looked okay wrt. smart, and I mostly just wanted to write about it:

[lsc@mcgrigor ~]$ sudo iostat -x /dev/sda /dev/sdb /dev/sdc /dev/sdd Linux 2.6.18-371.3.1.el5xen (mcgrigor.prgmr.com) 01/16/2014

avg-cpu: %user %nice %system %iowait %steal %idle

           0.00    0.00    0.05    0.02    0.00   99.93
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

sda 0.70 75.11 35.66 1.38 4568.62 611.67 139.85 0.36 9.61 0.53 1.95

sdb 0.46 75.10 35.62 1.39 4566.77 611.67 139.89 0.22 5.89 0.45 1.66

sdc 0.80 75.14 35.63 1.35 4569.63 611.63 140.10 0.64 17.18 0.57 2.10

sdd 0.46 75.09 35.62 1.40 4566.60 611.63 139.87 0.13 3.47 0.40 1.49

(this is a new server built out of older disks that appears to have the problem. It's not so bad that I get significant iowait when idle, but if you try to do anything, you are in a world of hurt.)

Check out the await value. re-do the same command with a '1' after /dev/sdd and it will repeat every second. If sdd consistently has a much worse await, it is what is killing your RAID. Drop the drive from the raid. If performance is better, replace the drive. If performance is worse (and with raid z2, it should be worse if you killed the drive) the drive was fine.

(Of course you want to do the usual check with smart and the like before this)

The interesting part of this failure mode that I have seen is that /throughput/ isn't that much worse than healthy. You get reasonable speeds on your dd tests. but latency makes the whole thing unusable.

Re: Tell HN: Server Status

#116

Post by lsc »

JoeAltmaier wrote: post #39
barrkel wrote: post #35

By tolerating the loss of two disks, do you mean raidz2 or do you mean 3-way mirror? Raidz2 is not fast. In fact, it is slow. Also, it is less reliable than a two way mirror in most configurations, because recovering from a disk loss requires reading the entirety of every other disk, whereas recovering from loss in a mirror requires reading the entirety of one disk. The multiplication of the probabilities don't work…

Reading 2 disks is not very much slower than 1. Its always the I/o bandwidth that is at issue, so it very much depends on the connectivity. And it all pretty much impacts your customer. We've all suffered under the 4-hour mirror rebuild, the whole machine made inoperable by the constant disk load. The only way to alleviate that, is to design extra bandwidth (e.g. another cable and controller) that's used exclusively…

>Reading 2 disks is not very much slower than 1. Its always the I/o bandwidth that is at issue, so it very much depends on the connectivity.

This was a problem in the old shared-bus u320 days... but now that we've got a 3 or 6 gigabit serial link to each disk? the bottleneck, unless you have some super-fancy SSD shit going on, is going to be getting the bits off the disk. I don't know of any spinning disk that can consistently saturate even a 2 gigabit link.

That's the thing... random access on spinning rust is staggeringly slow compared to almost everything else your computer does... and while a rebuild is sequential access, mostly, if you are trying to use the system during the rebuild? well, simultaneous sequential accesses become random access, so yea, your system is gonna suck during the rebuild anyhow. Add to this, well, disk diagnostics suck. Quite often a single disk will perform under-spec for some time before failing, slowing down the whole raid.

But SATA solved almost all of the bus bottleneck issues when it comes to disks.

Re: Tell HN: Server Status

#117

Post by vacri »

shiftpgdn wrote: post #52
barrkel wrote: post #35

By tolerating the loss of two disks, do you mean raidz2 or do you mean 3-way mirror? Raidz2 is not fast. In fact, it is slow. Also, it is less reliable than a two way mirror in most configurations, because recovering from a disk loss requires reading the entirety of every other disk, whereas recovering from loss in a mirror requires reading the entirety of one disk. The multiplication of the probabilities don't work…

Why not Raid 60 with Btrfs? It'll tolerate two disk loss with pro-active parity protection via btrfs and be faster and provide you with more disk space.

For a home NAS intended to stream video, my experience is 'no, don't'. I ran btrfs on my home NAS (on mdadm RAID1), and playing video from it was awful - pausing every few minutes. I thought it was just my misconfiguration of samba, but a rebuild using xfs fixed it, and a friend having the same problems on a btrfs (no raid) home media server now make me gun-shy. I don't need the benefits btrfs has on my box, so I'll stick with xfs.

I am very much not a filesystem expert, but my experience is that for me, btrfs is no benefits and significant problems.

Re: Tell HN: Server Status

#118

Post by vacri »

trout wrote: post #47
cincinnatus wrote: post #6

I'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.

If HN was on AWS, where would we go to discuss AWS outages?

Just don't put it in AWS US East, and you can talk about AWS outages just fine.

Re: Tell HN: Server Status

#119

Post by OafTobark »

Harj wrote: post #98
catinsocks wrote: post #68

Earlier quoted context omitted.

HN is not a charity, it is a marketing platform for YC with some community aspects. There is a very strong bias to everything YC. The HN community has also outgrown the software HN was built on you can see this in threads like: https://news.ycombinator.com/item?id=7051091 but even that thread is an extreme many front page items that gain traction are hard to go through because of things like lack of foldable comments…

HN is not a charity, it is a marketing platform for YC with some community aspects. Feels more like a community with some YC marketing aspects to me.

I disagree. It feels very YC driven.

Re: Tell HN: Server Status

#120

Post by annnnd »

colinbartlett wrote: post #96
annnnd wrote: post #72

Earlier quoted context omitted.

Yes, but I don't think such shortages do them much harm.

Sadly, I agree. But only because there's not a viable alternative. It's like if YouTube goes down for an hour, everyone whines but really nothing changes.

I wouldn't say that. I think they are competitive because they focus on what is important and disregard the rest as much as they can. And high availability is clearly not what is important for success in this case. With Youtube I wouldn't be so sure...
Post reply on HN