I think that's true within a field, yeah. But outside the
really top conferences, stuff as famous as SIGGRAPH, my experience has been that publisher affiliation is given significant heuristic weight, especially when people slightly outside a field are evaluating a conference they aren't personally familiar with (either researchers in other areas, or administrators/deans/etc.). A non-HCI researcher (even another CS researcher) might wonder, is this HCI conference I've never heard of any good? If it's an ACM HCI conference, many people will automatically attribute to it some baseline level of credibility, while if it's a completely independent conference, you start out more in the hole having to justify that this conference is legit. A top conference like SIGGRAPH could probably get away with not being ACM (though the ACM affiliation probably helps with deans/etc.), but more specialized, mid-tier conferences have a harder time going DIY.
I don't think that should always be the decisive factor, just that it's an issue. Last year I was proceedings chair of a conference that I did help move out of ACM, which it had been affiliated with and published with for the previous few years (http://www.fdg2013.org). There were various reasons for it, one being that being ACM-affiliated imposes significant bureaucracy and some financial costs that have to be recouped in registration fees (they're super-paranoid about the event insurance, for example). Another is that the conference is in a field (game studies, game AI, etc.) where we hope non-academics will also read the papers, and being open access would make that a lot easier. Being in the ACM Digital Library doesn't prohibit authors from putting up non-paywalled PDFs on their own sites, but it does prevent there from being an official, centralized open-access proceedings. But I would say we did lose some previous participants as a result of the move. For some people, moving from "an ACM conference, with proceedings archived in the ACM Digital Library" to "an independent conference, with proceedings published by the Society for the Advancement of the Science of Digital Games" is a deal-breaker: they see it as a drop off in prestige, from something they can list on their CV and use for their tenure case to something they can't. This varies completely based on university and field, of course, but a number of people told us losing the ACM publication was a big deal for them, and would lead them to publish elsewhere. One researcher from Brazil even said that his travel funding depends on the conference being sponsored by a major professional organization (IEEE/AAAI/ACM/etc.), so if we went independent he wouldn't be able to come anymore. I still think it was the right move on balance, but I can see the situations some of these people are in as well.
There's also a little bit of an archival question. The ACM DL will probably still exist in some form in 50 years. Will an independent conference's archive still exist in 50 years? To tackle that question we're investigating being archived by a new open-access archival initiative from the University of California (http://escholarship.org). There's a possibility this may also mollify some of the people upset about losing the "ACM" publisher line, since they could instead list a "University of California" publisher line, which is also a well-known institution. Another conference I've published in has taken that strategy, with the University of Michigan (http://quod.lib.umich.edu/i/icmc/). I think this might be an attractive way for conferences to keep the institutionally-archived cred while getting out of the paywall mess. However it does require these archives to then do some decent vetting so they don't archive fake conferences and lose their credibility.
