I think part of the difficulty of knowing when (or when not) to quit is that opportunity lurks even inside failure.
My first startup attempt was a coaching marketplace and community for poker players. My co-founder and I were both successful professional poker players and coaches at the time, but we were young (20ish) and all but clueless about technology. I still think the market opportunity was there, but we hired an agency who outsourced everything, made bad tech choices, took forever, and gave us a fraction of what we originally agreed on. I learned to program myself in order to complete a major component of the site (in flash) which they had just completely botched.
What we ended up with wasn't terrible, but it fell way short of the vision we had, and our marketing efforts fell mostly flat as well. We were able to recruit a few smart people to post and discuss hands here and there, and we wrote a few articles ourselves that showcased the flash hand analysis tool I had built (the best functioning aspect of the site), but we weren't getting any other traffic. There were a few bugs and an assortment of minor ux issues which made the site feel a bit clunky, and people's feedback was pretty unenthusiastic, so all in all we weren't feeling very good about the direction of things. It didn't take very long (maybe six weeks or so) before we threw in the towel.
It wasn't a complete failure--after all, I had learned how to program and discovered that I quite liked it (certainly much more than grinding poker online), and both of us had learned a lot of lessons about business and technology. But there was another lesson that we only discovered with time.
Even after we completely abandoned the site and it went dormant, I would still check google analytics from time to time to see if there was any traffic, and an interesting trend started to emerge: a few of the hand posts and articles we had posted were developing really steady search traffic. One article in particular was generating something like 20 uniques per day.
At that point, we were both just done with the whole endeavor and had no motivation to try to pick it up again, but we realized that if we had kept at it and continued to create more and more quality content, that we probably could have had an seo goldmine on our hands, developed reputations as poker strategy bloggers, and leveraged that into the community we had originally envisioned. Who knows if it would have gone that way, but it seems like there was at least a relatively unobstructed path to success available to us if we had been willing to stick it out and let the long tail kick in.