I disagree, because I think the idiosyncratic aspect was way more of a feature than a bug of Pitchfork. The real purpose of Pitchfork was not to say which music was good or bad. It was to say which music is worth talking about, and it made itself the center of that conversation. Getting reviewed by Pitchfork was more important than getting a good review by Pitchfork. They were notorious for giving bad reviews to good music.
If you're familiar with pro wrestling / kayfabe, then Pitchfork was the heel. They provide something for fans to root against. Look at how much sympathy artists get from their fans when Pitchfork published a bad review. If they didn't think an album was worth talking about, they ignored it. It is worse to be ignored by Pitchfork than it is to get a 0.0 review consisting entirely of a video of a monkey urinating into his own mouth (Jet's 2006 "Shine On").
Music criticism had been corrupted and corporatized and in the way video game reviewing still is. Everything from the big labels got an above-average but not perfect score, so nothing really stood out. You couldn't really be all that critical. Music reviews were boring. Pitchfork shook that up precisely because they printed controversial reviews. And as you just admitted, that was there from the start, so it is not a reason for the fall.
You never knew what to expect out of Pitchfork, and that was why people followed it so closely. But nobody read Pitchfork the way HNers probably read Metacritic or Rotten Tomatoes. Neither reviewers or readers were trying to objectively identify the best music. The 0.0 to 10.0 rating is not intended to be a scalar vector or even quantitative; it is an opening to a conversation, expressed as a float.