Meta Analysis
Chapter 22 Meta-Analysis
‘Fett’s Law: Never replicate a successful experiment’
Content list What is meta-analysis? Examples of meta-analytic studies Conducting a meta-analysis Replication and meta-analysis Comparing studies by effect size Combining studies by effect size Comparing studies by significance levels Combining studies by significance levels Comparing and combining more than two effect sizes and significance levels Some issues in meta-analysis 533 535 536 539 540 542 544 545 547 547
By the end of this chapter you will understand:
1 What meta-analysis is. 2 How it helps in confirming research findings. 3 How to undertake a meta-analysis.
Introduction Meta-analysis has become an important research strategy as it enables researchers to combine the results of many pieces of research on a topic to determine whether the finding holds generally. This is better than trying to assume that the findings of a single study have global meaning.
What is meta-analysis?
Each strand of a rope contributes to the strength of the rope. But the rope is stronger than any individual strand. Similarly, when a particular finding is obtained repeatedly,
534
EXTENSION CHAPTERS ON ADVANCED TECHNIQUES
under a variety of conditions, we are strongly confident that there exists a general principle. The results of small localized individual studies, no matter how well conducted, are often insufficient to provide us with confident answers to questions of general importance. Meta-analysis allows us to compare or combine results across a set of similar studies. In the individual study, the units of analysis are the individual observations. In meta-analysis the units of analysis are the results of individual studies. The term meta-analysis means ‘an analysis of analysis’. A particular topic may have been replicated in various ways, using, for example, differently sized samples, and conducted in different countries under different environmental, social and economic conditions.
‘Fett’s Law: Never replicate a successful experiment’
Content list What is meta-analysis? Examples of meta-analytic studies Conducting a meta-analysis Replication and meta-analysis Comparing studies by effect size Combining studies by effect size Comparing studies by significance levels Combining studies by significance levels Comparing and combining more than two effect sizes and significance levels Some issues in meta-analysis 533 535 536 539 540 542 544 545 547 547
By the end of this chapter you will understand:
1 What meta-analysis is. 2 How it helps in confirming research findings. 3 How to undertake a meta-analysis.
Introduction Meta-analysis has become an important research strategy as it enables researchers to combine the results of many pieces of research on a topic to determine whether the finding holds generally. This is better than trying to assume that the findings of a single study have global meaning.
What is meta-analysis?
Each strand of a rope contributes to the strength of the rope. But the rope is stronger than any individual strand. Similarly, when a particular finding is obtained repeatedly,
534
EXTENSION CHAPTERS ON ADVANCED TECHNIQUES
under a variety of conditions, we are strongly confident that there exists a general principle. The results of small localized individual studies, no matter how well conducted, are often insufficient to provide us with confident answers to questions of general importance. Meta-analysis allows us to compare or combine results across a set of similar studies. In the individual study, the units of analysis are the individual observations. In meta-analysis the units of analysis are the results of individual studies. The term meta-analysis means ‘an analysis of analysis’. A particular topic may have been replicated in various ways, using, for example, differently sized samples, and conducted in different countries under different environmental, social and economic conditions.
References: Cooper, H. & Rosenthal, R. 1982. Statistical versus traditional methods for summarising research findings. Psychological Bulletin, 87, 442–449. De Dreu, C.K. & Weingart, L.R. 2003. Task versus relationship conflict, team performance and team member satisfaction: a meta analysis. Journal of Applied Psychology, 88 (4), 741–749. Gully, S., Incalcaterra, K., Joshi, A. & Beaubien, J. 2002. A meta analysis of team efficiency, potency and performance. Journal of Applied Psychology, 87 (5), 819–832. Hosada, M., Stone-Romero, E. & Coats, G. 2003. The effects of physical attractiveness on job related outcomes. Personnel Psychology, 56 (2), 431–448. Iaffaldano, M. & Muchinsky, P.M. 1985. Job satisfaction and job performance: a meta analysis. Psychological Bulletin, 97, 251–273. Jenkins, J. 1986. Financial incentives. In Generalising from Laboratory to Field Settings. Locke, E. (ed). Lexington: Lexington Books. Judge, T.A., Colberet, A. & Illies, R. 2004. Intelligence and leadership. Journal of Applied Psychology, 89 (3), 542–552. Mullen, B. & Copper, C. 1994. The relation between group cohesion and performance. Psychological Bulletin, 115 (2), 210–227. Mullen, B. & Rosenthal, R. 1985. Basic Meta-analysis. Hillsdale: Lawrence Erlbaum. Rosenthal, R. 1979. The file drawer problem. Psychological Bulletin, 86, 638–641. Rosenthal, R. 1984. Meta analytic procedures for social research, in Applied Social Science Research Methods, Vol 6. Beverly Hills: Sage. Rosenthal, R. 1994. Interpersonal expectancy effects. A 30 year perspective. Current Directions in Psychological Science, 3, 176–179. Rosenthal, R. 1991. Meta-analytic Procedures for Social Research. Newbury Park: Sage. Smith, M. & Glass, G. 1977. Meta-analysis of psychotherapy outcome studies. American Psychologist. 32, 752–760. Thorsteinson, T.J. 2003. Job attitudes of part-time vs. full-time workers. Journal of Occupational and Organisational Psychology, 76 (2), 151–177. You will never have to undertake meta-analysis by hand. There are computer programs for this. For example: 1 Meta-Stat – A Tool for the Meta-Analysis of Research Studies. Produced by Lawrence M. Rudner, Gene V. Glass, David L. Evartt and Patrick J. Emery. Meta-Stat is a DOSbased computer program that automates the many complex tasks that are required to META-ANALYSIS 551 2 perform a meta-analysis. The data can easily be output in a format ready for use by SPSS. Meta-Stat is free for non-commercial, educational use. Meta-Stat is available through the auspices of the ERIC Clearinghouse on Assessment and Evaluation, Department of Measurement, Statistics and Evaluation, University of Maryland, College Park. Comprehensive Meta-Analysis Program (CMA) produced by Biostat. A free trial download is available at www.power.analysis.com/about.biostat.htm Now turn to the website page for this chapter and undertake the activities there.