The Public Relations Society of America promised in May that a final report of its 2011 member survey would be made available to its membership “later this summer.” But when results of the survey were announced recently by PRSA President and CEO Rosanna Fiske, what was presented was her version of the survey results.
The actual results were nowhere to be found.
An email was sent to PRSA asking where the actual results were. PRSA originally refused to release the results, citing competitive reasons, even though its 2008 survey results had been available online. A day later, however, PRSA posted the 2011 results on its PRSAY blog.
The results, as presented, contradicted Fiske’s otherwise glowing review what members appeared to be thinking.
Only 56 percent of the survey respondents indicate “satisfaction with membership.” To those of us who attempt to practice ethical PR, an obvious emphasis should be on improving upon whatever is leading to 44 percent of members being less than satisfied.
Instead, potentially negative information wasn’t mentioned either in the results or in Fiske’s blog post. More to the point, a basic calculation about the number of PRSA members who responded to the survey shows that very little should have been inferred from the survey in the first place.
The reason why has to do with basic survey methodology.
PRSA’s survey error
PRSA wrote that “Ketchum worked with Braun Research to complete online interviews with: 1,126 current members, 202 lapsed members, and 584 never members.” It is unclear how these members were chosen and how well they represent the total membership, two points which PRSA recommends doing as best practices when reporting survey results. How members were chosen and how many responded are critical issues in determining the validity of the responses, as well as the accuracy of claims by PRSA leaders about the survey results.
Presumably, however, these interviews were the online surveys solicited to all members in June. PRSA says there are more than 21,000 members. If all members were surveyed, that means the response rate was about 5 percent*. More below…
From PRSA’s Code of Ethics
DISCLOSURE OF INFORMATION
Core Principle Open communication fosters informed decision making in a democratic society.
Intent:
To build trust with the public by revealing all information needed for responsible decision making.
Guidelines:
A member shall:
- Be honest and accurate in all communications.
- Act promptly to correct erroneous communications for which the member is responsible.
- Investigate the truthfulness and accuracy of information released on behalf of those represented.
- Reveal the sponsors for causes and interests represented.
- Disclose financial interest (such as stock ownership) in a client’s organization.
- Avoid deceptive practices.
It is likely PRSA experienced a sampling error with its recent member survey. The problem is that an adequate response rate is subject to debate. (In my graduate survey research course, we were told we needed at least a 70-percent response rate in order to draw a reliable inference from survey results.) Nevertheless, this source discusses how to handle a low response rate.
A presumed 5-percent response rate* should be considered low; therefore, any inferences drawn from this survey should be made with a caveat that a low-response rate means it is difficult to generalize the results across the entire PRSA membership.
That’s not what PRSA did.
PRSA’s claims
PRSA choose instead to boast the following:
“…Our research shows that PRSA members not only value the National organization and their local Chapters almost equally, but also appreciate the National organization just a tiny bit more.” – William Murray, PRSA president and COO
“(Members) are … incredibly satisfied with the value of their PRSA membership.” – Rosanna Fiske, PRSA chair and CEO
PRSA’s response
When criticized for spinning the survey results, PRSA went on the offensive. Arthur Yann, APR, vice president of communications, accused this author of smearing the society and criticized the approval of a post comment by longtime PRSA critic Jack O’Dwyer, by saying that “you continue to entertain offensive, discriminatory, possibly libelous and certainly non-germane comments on your blog posts.”
Yann also accused this blog of providing a “selective interpretation” of the data even though the original post ended with this statement: “It is important to note that because of how the results are presented, more than one interpretation can be made…. I post my analysis to provide a perspective I believe is lacking in PRSA’s versions of the results.”
Fiske responded most reasonably, saying, “Any research needs greater context in order to synthesize the data. Providing our different perspectives here is part of that process.”
Indeed.
What is troubling is PRSA’s adherence to a perspective that seeks only to bolster the image of PRSA. At a time – it’s ethics month – when the society is pushing for ethical practice, and rightfully calling to the carpet other organizations that suffer from ethical lapses – including the PR firm, Ketchum, that was contracted to do the PRSA survey – PRSA makes great effort to diminish, ignore and divert attention away from its own transgressions.
To call this disingenuous is to be charitable.
* PRSA said it would make other results available by request: “PRSA members may request access to specific data from the survey by sending an email prsay@prsa.org.” An email sent a week ago to three of PRSA’s communications personnel requesting the actual survey response rate, among other questions. The information has yet to be provided. This post will be updated if it is.
Bill Sledzik
8 months ago
Enjoyed the follow-up post, Bob. I wonder if PRSA would send me the data, given that I was a participant in the study (one of the 202 lapsed members). I’d ask, but since PRSA has not responded to the request from you, a member in good standing, I’d just be wasting band width.
For what it’s worth, while still a member I once sent a similar request for a copy of the transcripts of an Assembly meeting. I was interested in what our national governing body was up to. I was, in fact, told directly by a PRSA employee it was available to members.
That request, like yours, never got a response.
While I doubt PRSA has a legal obligation to release this data, it does have an ethical obligation to let members to see what’s behind the curtain. It’s the only way to judge for ourselves if the PRSA interpretation is one we should trust. That data must include details on methodology.
PRSA also has an obligation not to “spin” the information. Yes, one can interpret data in many ways. That’s true. But a fair and balanced interpretation is what we need here, not cheerleading from HQ.
I remain most troubled by the defensiveness of an organization that preaches transparency in its core principles. That defensiveness is evident in comments to your earlier post, and it’s out of line.
Your interpretation of the PRSA Code of Ethics is spot on. While this is hardly huge scandal for the profession, it is a clear case of spinning the data in a way that, as you put it, seeks only to bolster the image of PRSA.
No one will benefit from this survey. But if, in fact, all 21,000+ members were asked to participate, no one cares either.
Bob
8 months ago
Thanks, Bill. PRSA personnel hinted they needed more time than my previous deadline, which was about a day and half. When I asked for an ETA, I received no response. So I waited nearly a week before posting this.
It seems to me at any point they could have supplied some, all or none of the information and simply explained why. The same folks are all over the ethics chat on Twitter this week, which appears to be the priority — to chat about ethical principles rather than to model them.
I make attempts to avoid being hyperbolic (don’t laugh), but I’m not sure how else to describe all of this. I think in their minds, if criticisms are raised — which may echo some of O’Dwyer’s — then it must be easy to paint those folks with the same brush they apply to O’Dwyer and to simply blow them off.
Ann VerWiebe
8 months ago
I am not a PRSA member, although I generally enjoy the programs presented by the several chapters in my area. The added cost of attending programs at the non-member rate doesn’t make up what it would cost for the actual membership (it’s expensive for someone, like me, who works for a non-profit). And, I’ve never really been recruited, even though I am a public relations professional. That said, 70% is a really high number for survey respondents. It seems to me that you could only pull that off if everyone was already highly invested in the results – positive or negative. When I was in grad school, they used a 20 – 30% benchmark for responses (5% is just sad – and this was a survey sent to people who understand the value of surveys). The low response rate is making a statement on its own – that the membership doesn’t care enough to even comment on their reactions.
Bob
8 months ago
Thanks for your comment, Ann. The response rate issue is one beyond my expertise, so I tend to defer to one of my mentors, who studied under survey expert Don Dillman, and who stressed the 70% response rate for graduate-level research. Others indicate much lower rates are acceptable. This source indicates as low 18% is acceptable depending on the level of accuracy required: http://www.greatbrook.com/survey_statistical_confidence.htm.
In any case, it’s important to note that WE DON’T KNOW PRSA’s survey response rate because its representatives won’t tell us. I’ve asked twice now. The 5% rate I mention is a calculation of their total membership versus the number of responses PRSA indicated it received.
Our local chapter usually gets about a 10-20% response rate. The results are used for general feedback about programs. Since I report the survey results to the board, I’ve mentioned the results should be considered anecdotal and not representative of the total number of attendees for our programs. The results are useful, but I would also caution against making grandiose claims about them.