The citizen and media-friendly website, Public Agenda, has handy guides for understanding surveys, polls and the nuances that go into what constitutes scientific polling. This guide for journalists has 20 questions that should be asked about surveys. Among them is this: How were (the survey participants) chosen?

Amazingly, a randomly sampled population means that only about 1,000 people can be surveyed, or polled, in order to generalize the survey’s results across “more than 210 million American adults.”

The key word is random. Not just any survey with 1,000 respondents will yield the same level of confidence in the results. This is a small but critical point the Public Relations Society of America is muddying with its presentation of the Society’s recent member survey results. (Read here and here for parts one and two in this series.)

In a recent post on the PRSAY blog, David Rockland of Ketchum, PRSA’s PR agency contracted to conduct its surveys, insists that PRSA’s survey methodology is sound. What Rockland and PRSA leaders have not presented is the importance of a random sample versus what PRSA did, which was to sample all 21,000 PRSA members.

The consequence is that the most likely survey errors experienced by PRSA are a non-response error, described as “the bias introduced when individuals invited to take the survey do not take the survey,” and self-selection bias.

 “Respondents who volunteer to participate in such surveys tend to be more extreme or otherwise very different in their views than those who do not. In no way can they be said to be representative of the population, so the survey results cannot be used to say anything useful about a target population.”

PRSA will not reveal the survey’s response rate (nor will it say how much it paid Ketchum to conduct the survey). A calculation of total membership versus those who took the survey shows a likely response rate of about 5 percent.

Rockland says the respondents were “1,126 current members, 202 lapsed members and 584 people who have never been a member. To put that into perspective, most surveys you see in the news have a sample size of 1,000 for the entire American public. Results of this study are projectable to the overall populations within the respective margins of error at the 95-percent confidence level.”

PRSA extrapolated its survey results to its entire membership, saying that “responses were weighted to the overall profile of the PRSA membership in terms of tenure in the PR industry. This is to ensure results approximate the membership as closely as possible, and is a standard practice in survey research.”

Survey experts are clear that higher response rates are only needed up to a point, if the actual people surveyed are randomly selected. PRSA put a call out to its entire membership and surveyed them online, another variant that can skew the validity of the responses. Surveying an entire population and then receiving a 5-percent response rate puts the overall responses in doubt since it is unlikely that 5 percent represents the entire PRSA membership.

The American Association for Public Opinion Research calls this SLOP (self-selected listener opinion poll) polling: “Respondents who volunteer to participate in such surveys tend to be more extreme or otherwise very different in their views than those who do not. In no way can they be said to be representative of the population, so the survey results cannot be used to say anything useful about a target population.”

Rockland disagrees: “No matter how you slice the data, or quibble with one number or another, the overall finding of ‘significant improvement over 2008 and high current levels of member engagement’ holds true.” (It should be noted that significance in research is a deliberate statistical term often confused by the lay public as meaning important. Rockland’s use of the term is unclear.)

He also says that whatever bias PRSA saw in its survey results was also in the 2008 survey results, but that the change between surveys is what’s important: “…this bias would be as much the case in the study we did in 2008 as it would this year, so in terms of looking at changes, we are fine.” In other words, survey bias is less important because PRSA only needs to see change from one survey to the next.

Most telling, however, is when Rockland says that criticism of the survey interferes with progress. He writes: “… the worst thing one can do is spend time trying to find what could be right or wrong in the data, versus taking action in continuing to move PRSA in the positive direction it is going.”

PRSA would prefer that this positive direction not be muddied by criticism. When PRSA leaders make grandiose claims about satisfaction among its members, and fail to provide supporting information used to make those pronouncements – in contradiction to its own ethics code and best practices recommendations – the Society expects its members to accept these claims without question.

The net outcome of PRSA’s survey is that there are no problems evident in the results, only a few “opportunities for improvement.”