It's fine until your garage burns down / floods, or someone digs through your street's connection. Those things are unlikely, but even with a backup you're going to have pretty horrendous amounts of downtime. You could lessen the impact by having some off-site secondary system ready to go at any moment, but then at that point you might as well be renting a server.
More than calamities, I'd worry about simply going away for a week or two and having something happen. With Linode/whoever, even if I'm somewhere remote, I suppose I can at least call or write email and get some help. If everything's at my house, what would I do? Leave the keys to a friend? "Can you feed the cats and take care of the servers in case they have a problem at 3AM?"
LOL now this would be a friend.