David D. Williams
Evaluation is integral to every aspect of designing instruction with learning objects. Evaluation helps in clarifying audiences and their values, identifying needs, considering alternative ways to meet needs (including selecting among various learning objects), conceptualizing a design, developing prototypes and actual instructional units with various combinations of learning objects, implementing and delivering the instruction, managing the learning experience, and improving the evaluation itself. Evaluation must assemble all the standards associated with objects, learners, instructional theories, and other stakeholder values and estimate the quality of the instruction in terms of those standards both to formatively (for development purposes) improve the instruction and to assess its value summatively (for accountability purposes), as well as determining degrees of compliance with technical standards. This chapter summarizes current relevant evaluation theories and practical guidelines for building evaluation principles into the entire process of designing instruction with learning objects throughout the life of any given unit of instruction. It also emphasizes the need to include evaluation as an integral part of any design process by addressing the following questions:
1. What is evaluation? And what does it have to do with learning objects? What is the most current thinking about evaluation, particularly participant oriented and democratic evaluation? What are they and how do they fit with learning object and instruction evaluation needs? How does the world of values fit and not fit with the learning object world?
2. Who cares? Who will use the information gathered through an evaluation of particular learning objects and instruction using those objects?
3. What do they care about? Definitions. What are users' definitions of learning
References: Chelimsky, E. (1997). In E. Chelimsky and W. R. Shadish (eds.), Evaluation for the 21st century: a handbook. Thousand Oaks, CA: Sage. Cousins, J. B., Donohue, J. J., & Bloom, G. A. (1996). Collaborative evaluation in North America: evaluators ' self-reported opinions, practices, and consequences. Evaluation Practice 17 (3), 207-226. Cousins, J. B. & Whitmore, E. (1998). Framing participatory evaluation. In E. Whitmore (ed.), Participatory evaluation approaches. San Francisco: Jossey-Bass. Fetterman, D. M. (1996). Empowerment evaluation: knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage. Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation. Thousand Oaks, CA: Sage. House, E. R. and Howe, K. R. (1999). Values in education and social research. Thousand Oaks, CA: Sage. LTSC. (2000). Learning technology standards committee website [On-line]. Available: http://ltsc.ieee.org/ Patton, M Ryan, K. E. and DeStefano, L. (Eds) (2000). Evaluation as a democratic process: promoting inclusion, dialogue, and deliberation. New Directions for Program Evaluation, no. 30. San Francisco: Jossey-Bass. Sanders, J. R. (Chair) (1994). The program evaluation standards. (2nd ed.) Thousand Oaks, CA: Sage. Scriven, M. (1980). The logic of evaluation. Inverness, CA: Edgepress. Scriven, M. (1991). Evaluation thesaurus. (4th ed.) Thousand Oaks, CA: Sage Stake, R Stufflebeam, D. L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education. 5(1), 19-25. Stufflebeam, D L. (1975). Metaevaluation. Occasional Paper Series, no. 3. Kalamazoo, Mich.: Evaluation Center, Western Michigan University. Wiley, D. A. (2000). Learning object design and sequencing theory. Unpublished Doctoral Dissertation, Brigham Young University, Provo, UT. Available: http://davidwiley.com/papers/dissertation/dissertation.pdf Worthen, B