Pilot Studies
Before testing the effect of technical detail on perceived risk, one must determine whether laypeople recognize technical detail as defined by experts; i.e., are lone paragraphs that contain more detail according to experts seen by lay readers as more detailed? When these paragraphs are combined in entire stories about the PERC spill, do readers distinguish the stories’ respective levels of detail? These issues are important because they affect the value of providing technical information to citizens. If the public cannot recognize expert-defined detail as indeed more detailed, whether in separate sentences, paragraphs or entire stories, presenting the data as being more detailed may backfire if citizens want details. They will believe their demand evoked no response and may react with anger. Alternatively, if readers do not notice greater detail, more information could be put into a story without people feeling that they are asked to do a lot of mental work.
Article Table of Contents
Eighty-six Rutgers University undergraduates read one of two versions (high or low on all three manipulation variables) of the one-page simulated news story. Then, without being allowed to refer to the story, they answered written questions about their overall reactions to the news stories. Finally, they rated the technical detail in three highlighted passages in a second copy of the mock news story.
In their reading of the entire story, subjects reading more and less detailed stories expressed no difference in amount of perceived detail in the story, clarity of information about the risk, or their resulting understanding of the nature and extent of the risk. There was no significant correlation between perceived risk and either actual or perceived level of detail.
When their attention focused on specific paragraphs, however, subjects who read more detailed paragraphs saw more detail about potential health effects and rated their understanding of health effects as higher. No significant differences were found for the equivalent measures about exposure routes. This part of the pilot study thus found that at least sometimes experts and laypeople can agree on which paragraphs are more detailed. The amount of detail in specific paragraphs was not significantly correlated with perceptions of the spill’s risk or seriousness, its likelihood to cause a health problem in the family, the anger it caused, or the importance of NJDEPE taking remedial action.
The level of variability in the measures and centrality of responses ruled out instrument biases as an explanation of these results. Moreover, NJDEPE experts judged that there was a substantial difference in the amount of technical detail contained in the news stories. In fact, the highly detailed version was judged to exceed the amount of detail in typical journalistic coverage of environmental spills. Thus this difference in perceived detail for the whole news story between citizens and experts seems to be a substantive finding rather than a methodological shortcoming.
A second pretest with 42 Idaho State University students was identical to the first, except it deleted the paragraph-rating task and added six items testing societal and personal risk aversion. Again the technical detail manipulation did not work; subjects’ ratings did not distinguish between the two stories in terms of the overall amount of detail. Controlling for risk aversion did increase the sensitivity of the tests of differences due to detail on some of the dependent variables. Significance increased from 0.05 to 0.01 for the affective measure of anger, and the cognitive measure of the risk’s “importance” went from non-significant to significant at 0.05. However, the correlations of technical detail with all measures of perceived risk remained non-significant even controlling for risk aversion.
The pilot studies suggested that merely providing more technical information does not make people see a news story as more detailed, nor alter their risk perceptions. However, given the small and collegiate character of these samples, these findings still needed to be tested in a larger field study. It could not be assumed that mature homeowners would react the same as college students. Far more important, even if citizens do not perceive technical detail as do experts, experimental manipulation of technical detail may still affect perceived risk. Greater detail could shift risk perceptions without readers being aware that there was more detail in the story they read. Three variables were thus of interest for the field study:
- technical detail as defined and varied by the researchers and NJDEPE
- detail as perceived by the subjects
- perceived risk (and its behavioral correlate, intention to test one’s well water).
Field Study
For the field test, we shortened the simulated news stories, and revised elements that seemed to confuse pretest subjects. The agency advisory committee also wanted additional changes in the stories to make them more technically accurate and realistically depict agency behavior. The survey instrument also was shortened to include several questions allowing responses on a Likert scale (Table 1 above [omitted]).
Time and budget constraints altered the original plan for a 2x2x2 design to vary each of the three treatment variables. Instead, five stories were used. Both outrage and technical detail were varied in high and low versions while keeping magnitude low. This provided four different stories: high outrage-high detail, high outrage-low detail, low outrage-high detail, and low outrage-low detail. A fifth story combined high magnitude with low outrage and low detail.
Subjects were 595 New Jersey middle-class residents (88% response from 676 contacts). A cluster sample was used to select households from four suburbs near Rutgers University that represented Middlesex County, i.e., not unduly full of students and professors. Interviewers alternated the type of subject asked for at each house by age (oldest vs. youngest over 18) and gender. The median age of the sample was 40 years; males were 45.5%; 29.6% had a high school education or less, and 16.6% had graduate school training. Sex and age were comparable to 1990 Census data (50% male; median age, 38), but recent data on education for these towns were not available. For the county as a whole, 53% of the population had up to a high school education and about 10% had graduate training. However, the county included a poor urban area not part of the sample area, so these data may underestimate the sample area’s mean education levels. In short, the study sample was somewhat more female and perhaps better educated than the sampled population, with about the same age distribution.
Half of the subjects received the story first, then a six-item survey instrument, and finally a risk aversion/demographic questionnaire. The other half received the questionnaire first, then the story, and finally the survey instrument. No order effects were found. All subjects were asked to return the story before receiving the survey to avoid any rereading of the story in search of “correct” answers.
Results indicated that the technical detail manipulation did not significantly affect any dependent measure, including both perceived risk and intention to test well water, either alone or in a multiple regression analysis.26 There was also no correlation between technical detail, as manipulated by researchers in different stories, and the manipulation check “perceived detail.” Pretest results were thus confirmed. Technical detail was not perceptible and had no effect on risk perceptions within a range from typical journalistic reporting of technical information, “low,” to the most detail an agency might reasonably expect a newspaper to include, “high.” In contrast, “outrage” had strong, and risk magnitude weak, effects on risk perception.
Perceived detail, however, significantly correlated with ratings of the risk as serious (r = 0.11, p<0.05). In the perceived risk multiple regression analysis, perceived detail was a statistically significant independent variable (Table 2 [omitted]). However, it contributed only 0.02 unique variance of a total adjusted squared multiple correlation of 0.25, the third independent variable to enter the regression after societal risk aversion and the perceived appropriateness of agency behavior. Perceived detail was not a significant factor in a multiple regression analysis for intention to test one’s well water which, overall, explained only 14% of the variance. These analyses indicated that unknown factors not included in the regressions explained most of the variance in perceived risk and intention to test.
Table 2 [omitted]
Perceived detail significantly correlated with the perceived appropriateness of agency behavior (r = 0.44, p<0.0001). It also was affected by the outrage manipulation. People who read high-outrage stories saw them as containing much less detail [(1,495) = 10.61, p < 0.01)] than did those who read low-outrage stories. Perhaps people concluded that proper agency behavior on other points implies sufficiently detailed information. The direction of causation here is speculative, however, since the research design could not assess temporal priority of variables. It is unclear, too, how much weight should be given any finding on perceived detail, a variable intended to measure the success of the technical detail manipulation but found to be unrelated to it. These results for perceived detail and perceived risk may seem to support Mazur’s hypothesis27 that people who see more information about a hazard will conclude that the risk must be more serious. Yet, the effect in this study was for perceived detail, not actual detail as in Mazur’s hypothesis, and societal risk aversion had a greater effect on perceived risk. Thus it is still unclear whether Mazur’s hypothesis, that overall media coverage even when positive increases perceived risk, can be extended to much narrower sharing of information, such as individual news stories.
Discussion
This is the first experimental proof that how an agency behaves (or at least is reported to behave) is at least as critical for public perceptions of risk and agency performance as what the agency says or is reported to say.28 Outrage had a strong effect on risk perception through its shaping of perceived appropriateness of agency behavior, although it was not significant in the regression analysis unless perceived appropriateness was removed. In contrast, even five orders of magnitude of variance in risk magnitude had only modest effects on perceived risk, and there were no observable effects of technical detail. Perceived detail affected perceived risk, but far less than societal risk aversion and perceived appropriateness (Table 2 [omitted]). Furthermore, the significance of this relationship is unclear: perceived detail is not an obvious variable in actual environmental problems and debates among citizens and officials. Thus, the hypothesis that providing technical detail to citizens reduces perceived risk and increases perceived appropriateness of agency behavior received no support.
These results do not mean that transmission of technical information never has an effect. Clearly the confounding issues discussed earlier need to be explored in further research. These include the effects of different kinds of information; detail vs. story length; alarming or reassuring aspects of technical information; uncertainty; source credibility; channel effectiveness, e.g., press vs. television vs. personal interaction; and clarity of information. Information about PERC’s carcinogenic potential appeared in this study’s mock news stories. The study results suggest that alarm caused by such information, if any, can be reduced by “appropriate” agency behavior, although this needs further study. The strong effect of perceived appropriateness of agency actions on perceived risk suggests that the credibility of a given source may vary according to the messages it sends. It is not clear from this study whether agency credibility in the low outrage stories was due to the safety message contradicting expected messages from that source, as suggested earlier. For example, in the low-outrage condition the agency was willing to arrange further water testing and consider revising its standards; one result was lower perceived risk. The results may indicate that a government message can be credible because it treats citizen concerns as worthy of respect, rather than credibility stemming only from an admission that a situation is dangerous. These conflicting hypotheses must be tested directly.
When subjects in one pilot test were asked to assess the level of detail in paragraphs about health effects and exposure, they identified a difference in the first case but not the second. This different response to information on health effects and exposure routes may have been due to other differences between these topics. One study found that a large proportion, although not a majority, of the public does not recognize exposure as a necessary middle step in the process from pollutant emissions to health effects.29 This view may make exposure routes less salient than health effects, and thus technical details about exposure less noticeable.
The perception of detail in individual paragraphs but not in entire stories in one pilot test also raises another issue. Suppose this lay inability to perceive differences in technical detail could be generalized across other topics, audiences, and information types and channels. As suggested earlier, this could result in either greater outrage, because government or other information sources seem unresponsive to requests for more information, or greater willingness to read technical details, because this reading does not seem demanding. These alternative hypotheses have important consequences for risk communication if they are true, so research on them seems needed.
Although the lack of effect found for technical detail variables considered jointly suggests a lack of individual effect, this is not conclusive. Different components of technical information may offset each other. For example, the information about possible health effects in both high- and low-detail stories may have aroused alarm, while the information that exposure routes posed no immediate problem may have allayed concern, assuming that people take exposure into account–a questionable assumption as noted earlier.
Another issue for further research is whether the problem is not so much that technical information is ineffective as that the wrong information is being provided (or recommended) by hazard managers. For example, studies have found that citizens stress facts other than mortality, e.g., catastrophic potential, transgenerational effects, and perceived scientific knowledge, in characterizing a hazard.30 Providing technical information on these items might be more effective in addressing underlying public concerns than discussion of such mortality-related issues as exposure or health effects, although there is no evidence in favor of this hypothesis at present.
Conclusion
Many agency staff, corporate executives, and academic experts feel that giving citizens more detail about health effects data and likely exposure routes will reduce their concerns about low-probability risks. This study, although not definitive, suggests that providing more technical detail may not be the most useful strategy for risk communicators to pursue. In theory, both agency process (outrage) and science (technical detail) should help the public decide what risks are of concern. Yet officials who try to educate citizens on technical issues without also considering changes in their own behavior, e.g., more swiftly informing citizens how they are dealing with a problem and fully addressing public concerns, may do themselves a disservice.
Further efforts to explore the effects of technical information and its confounding variables on perceived risk could confirm how institutions can best address public concerns and the need for environmental education. Research on the causes and effects of perceived detail would help us understand what “useful technical information” means to citizens, as opposed to what it means to experts who try to provide it.
Copyright © 2001 by Branden B. Johnson, Peter M. Sandman, and Paul Miller
Part 1: Introduction, Possible Roles of Technical Information, and Research Design
Part 2: Pilot Studies, Field Study, Discussion, and Conclusion
