It’s an exciting day for word slingers around the world: The 2018 edition of The Associated Press Stylebook and Briefing on Media Law goes on sale this morning! As with previous editions, this update includes several hundred new and revised entries. This year, though, journalists and editors will also find a new “Polls and Surveys” chapter to provide guidance on using and reporting on surveys. According to an AP press release,
Stylebook editors worked with AP Deputy Managing Editor for Operations David Scott and AP Polling Editor Emily Swanson and consulted with Pew Research Center and NORC at the University of Chicago to review polling guidance and create a new chapter for the 2018 edition, which became available to AP Stylebook Online subscribers in April.
That new chapter opens with an important reminder — or perhaps warning — about the perilous nature of polls: “Reporting on public opinion research requires rigorous inspection of a poll’s methodology, provenance and results. The mere existence of a poll is not enough to make it news.” The second sentence in particular should, I believe, be Rule No. 1 for reporting on polls.
The largest portion of this new chapter’s 5 1/2 pages is devoted to exploring the most important question one must answer to evaluate a poll: Does it reflect the opinions of the population it purports to represent? For a poll to achieve a “yes,” the AP Stylebook states that it must
- “Disclose the questions asked, the results of the survey and the method in which it was conducted.” Other statisticians should be able to use the same data and draw the same conclusions.
- “Come from a source without a stake in the outcome of its results.” Conflicts of interest, or even the appearance of conflicts, can call survey conclusions into doubt.
- “Scientifically survey a random sample of a population, in which every member of that population has a known probability of inclusion.” Creating a truly random sample of respondents does not come easily, and poll results could be skewed if proper rigor was not taken.
- “Report the results in a timely manner.” Numbers change over time, and opinions can swing widely based on new evidence, world events, and controversial social media posts. The fresher a survey is, the more accurate it is likely to be.
The AP reminds readers that polls are not perfect, and so a survey’s margin of error ought to be reported (in percentage points not percent) along with the story, but even that does not account for all of a poll’s imperfections. Sampling error, the AP Stylebook notes, is
merely the only [error] that can be quantified using established and accepted statistical methods. Among other potential sources of error: the wording and order of questions, interviewer skill and refusal to participate by respondents randomly selected for a sample. As a result, total error in a survey may exceed the reported margin of error more often than would be predicted based on simple statistical calculations.
This applies to scientifically conducted polls; of a more dubious ilk are online surveys. The AP warns readers twice in this chapter, and using identical language, that “[t]here is no single established method of estimating error for surveys conducted online among people who volunteer to take part in surveys. These surveys are still subject to error, uncertainty and bias.”
The new “Polls and Surveys” chapter also touches on how to report poll results without bias, the dangers of comparing polls that use different methodologies, and the benefits to transparecy of including a stand-alone statement about polling methods.
We’ve all experienced survey results that have been twisted to say almost anything. The AP’s new guidance is a great step toward unraveling suspicious results and getting at the truth of poll results. Truthful and trustworthy reporting relies on understanding what polls and surveys can and cannot tell us. This new chapter of the AP Stylebook is a step in the right direction not only for journalists and editors, but for the public who rely on them to understand the world.
I’ll be writing about other changes in the recent update again next week.
Cover image courtesy The Associated Press.