Statistical Significance
Journal of Counseling Psychology 1983, Vol. 30, No, 3,459-463
Copyright 1983 by the American Psychological Association, Inc.
Statistical Significance, Power, and Effect Size: A Response to the Reexamination of Reviewer Bias
Bruce E. Wampold
Department of Educational Psychology University of Utah
Michael J. Furlong and Donald R. Atkinson
Graduate School of Education University of California, Santa Barbara
In responding to our study of the influence that statistical significance has on reviewers ' recommendations for the acceptance or rejection of a manuscript for publication (Atkinson, Furlong, & Wampold, 1982), Fagley and McKinney (1983) argue that reviewers were justified in rejecting the bogus study when nonsignificant results were reported due to what Fagley and McKinney indicate is the low power of the bogus study. The concept of power is discussed in the present article to show that the bogus study actually had adequate power to detect a large experimental effect and that attempts to design studies sensitive to small experimental effects are typically impractical for counseling research when complex designs are used. In addition, it is argued that the power of the bogus study compares favorably to that of research published in the Journal of Counseling Psychology at the time our study was completed. Finally, the importance of considering statistical significance, power, and effect size in the evaluation of research findings is discussed. Fagley and McKinney (1983) argue that we were not justified in concluding from our study of the manuscript reviewing process (Atkinson, Furlong, & Wampold, 1982) that a reviewer bias favoring statistically significant results was operating. The essence of their argument is that the power of the bogus study used as the manuscript stimulus in our design was, in their opinion, low (.79, .37, and .09 for large, medium, and small effects, respectively), and consequently reviewers were justified in rejecting it under
Copyright 1983 by the American Psychological Association, Inc.
Statistical Significance, Power, and Effect Size: A Response to the Reexamination of Reviewer Bias
Bruce E. Wampold
Department of Educational Psychology University of Utah
Michael J. Furlong and Donald R. Atkinson
Graduate School of Education University of California, Santa Barbara
In responding to our study of the influence that statistical significance has on reviewers ' recommendations for the acceptance or rejection of a manuscript for publication (Atkinson, Furlong, & Wampold, 1982), Fagley and McKinney (1983) argue that reviewers were justified in rejecting the bogus study when nonsignificant results were reported due to what Fagley and McKinney indicate is the low power of the bogus study. The concept of power is discussed in the present article to show that the bogus study actually had adequate power to detect a large experimental effect and that attempts to design studies sensitive to small experimental effects are typically impractical for counseling research when complex designs are used. In addition, it is argued that the power of the bogus study compares favorably to that of research published in the Journal of Counseling Psychology at the time our study was completed. Finally, the importance of considering statistical significance, power, and effect size in the evaluation of research findings is discussed. Fagley and McKinney (1983) argue that we were not justified in concluding from our study of the manuscript reviewing process (Atkinson, Furlong, & Wampold, 1982) that a reviewer bias favoring statistically significant results was operating. The essence of their argument is that the power of the bogus study used as the manuscript stimulus in our design was, in their opinion, low (.79, .37, and .09 for large, medium, and small effects, respectively), and consequently reviewers were justified in rejecting it under
References: Atkinson, D. R., Furlong, M. J., & Wampold, B. E. Statistical significance, reviewer evaluations, and the scientific process: Is there a (statistically) significant relationship? Journal of Counseling Psychology, 1982,29,189-194. Bradley, J. V. Distribution-free statistical tests. . Englewood Cliffs, N.J.: Prentice-Hall, 1968. Bradley, J. V. Robustness? British Journal of Mathematical and Statistical Psychology, 1978,37, 144-152. Cohen, J. Statistical power analysis for the behavioral sciences (Rev. ed.). New York: Academic Press, 1977. Craig, J. R., Eison, C. L., & Metze, L. P. Significance tests and their interpretation: An example utilizing 463 published research and o>2. Bulletin of the Psychonomic Society, 1976, 7, 280-282, Fagley, N, S., & McKinney, I. J. Reviewer bias for statistically significant results; A reexamination. Journal of Counseling Psychology, 1983, 30, 298-300. Garfield, S. L. Research problems in clinical diagnosis, Journal of Consulting and Clinical Psychology, 1978, 46,596-607. Greenberg, L. S., & Dompierre, L. M. Special effects on Gestalt two-chair dialogue on intrapsychic conflict in counseling. Journal of Counseling Psychology, 1981,28,288-294. Greenwald, A, G. Consequences of prejudice against the null hypothesis. Psychological Bulletin, 1975, 82,1-20. Haase, R. F., Waechter, D. M., & Solomon, G. S. How significant is a significant difference? Average effect size of research in counseling psychology. Journal of Counseling Psychology, 1982,29,58-65. Hays, W. L. Statistics for the social sciences (2nd ed.). New York: Holt, Rinehart & Winston, 1973. Kraemer, H. C. Coping strategies in psychiatric clinical research. Journal of Consulting and Clinical Psychology, 1981,49,309-319. LeVine, E., & Franco, J. N, A reassessment of selfdisclosure patterns among Anglo Americans and Hispanics. Journal of Counseling Psychology, 1981, 28,522-524. Mitchell, C., & Hartmann, D. P. A cautionary note on the use of omega squared to evaluate the effectiveness of behavioral treatments. Behavioral Assessment, 1981,3,93-100. O 'Grady, K. E, Measures of explained variance: Cautions and limitations. Psychological Bulletin, 1982,92,766-777. Received November 22,1982 Revision received February 14,1983 •