Case Incident 2

A Review of Factors Influencing User Satisfaction in Information Retrieval
Azzah Al-Maskari Directorate General of Technological Education P.O.Box: 413, Post code 100 Tel: [phone] Fax: [phone] Email: [e-mail] Mark Sanderson Department of Information Studies, University of Sheffield Regent Court, 211 Portobello Street Sheffield S1 4DP, UK Tel: [phone] Fax: [phone] Email: [e-mail]

1

A Review of Factors Influencing User Satisfaction in Information Retrieval
Abstract This paper investigates factors influencing user satisfaction in information retrieval. It is evident from this study that user satisfaction is a subjective variable which can be influenced by several factors such as system effectiveness, user effectiveness, user effort and user characteristics and expectations. Therefore, information retrieval evaluators should consider all these factors in obtaining user satisfaction and in using it as a criterion of system effectiveness. Previous studies have conflicting conclusion on the relationship between user satisfaction and system effectiveness, this study has substantiated this relationship and supports using user satisfaction as a criterion of system effectiveness. 1. Introduction

The main aim of an information retrieval (IR) system is to satisfy the need of its users. Lancaster (1979; 1981) suggests that an IR system can be evaluated according to three criteria: (i) the suitability of a system in terms of the specific IR tasks for which it will be used; (ii) the system‟s task performance efficiency and (iii) the extent to which the system satisfies the information needs of its users. Therefore user satisfaction is an important factor in evaluating IR systems. User satisfaction in IR research is generally considered a criterion of system success and effectiveness. Griffiths et al. (2007) surveyed the information retrieval and information system (IS) literature in an attempt to understand what constitutes

References: Al-Maskari, A., Sanderson, M. & Clough, P. (2008)."The Good and the Bad System: Does the Test Collection Predict Users‟ Effectiveness". In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp.59-66 Singapore: ACM. Baeza-Yates, R. & Ribeiro-Neto, B. (1999) Modern Information Retrieval, Boston, MA, Addison Wesley Longman Belkin, N. J. & Vickery, A. (1985) Interaction in information systems: a review of research from document retrieval to knowledge-based systems, London, British Library. Bruce, H. (1998) User satisfaction with information seeking on the Internet. Journal of the American Society for Information Science, 49 (6), 541-556. Cooper, W. S. (1968) Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19 (1), 30–41. Cooper, W. S. (1973) On selecting a measure of retrieval effectiveness part II. Implementation of the philosophy. Journal of the American Society for Information Science, 24 (6), 413-424. Egan, D. E. (1988) Individual differences in human-computer interaction. IN HELANDER, M. G., LANDAUER, T. K. & PRABHU, P. V. (Eds.) Handbook of human-computer interaction Amsterdam, Holland, Elsevier. Frøkjær, E., Hertzum, M. & Hornbæk, K. (2000)."Measuring Usability: Are Effectiveness, Efficiency, and Satisfaction Really Correlated?" In: Proceedings of the SIGCHI conference on Human Factors in computing systems, pp.345 - 352 The Hague, Amsterdam: ACM. Garoufallou, E., Trohopoulos, I., Siatri, R. & Balatsoukas, P. (2007)."A user-centred approach to the evaluation of digital cultural heritage maps: the VeriaGrid system ". In: Libraries Without Walls 7, Aegean Island of Lesvos, Greece: Gatian, A. W. (1994) Is user satisfaction a valid measure of system effectiveness? Information and Management, 26 (3), 119-131. Gluck, M. (1996) Exploring the relationship between user satisfaction and relevance in information systems. Information Processing and Management, 32 (1), 89-104. Griffiths, J., Johnson, F. & Hartley, R. (2007) User satisfaction as a measure of system performance. Journal of Librarianship and Science, 39 (3), 142-152. Harter, S. P. & Hert, C. A. (1997) Evaluation of information retrieval systems: Approaches, issues, and methods. Annual Review of Information Science and Technology (ARIST), 32, 3-94. Hersh, W., Buckley, C., Leone, T. J. & Hickam, D. (1994)."OHSUMED: an interactive retrieval evaluation and new large test collection for research". In: Proceedings of the annual international ACM SIGIR conference on Research and development in information retrieval pp.192-201. Dublin, Ireland: Springer-Verlag Hersh, W. & Hickam, D. H. (1995) An evaluation of interactive Boolean and natural language searching with an online medical textbook. Journal of the American Society for Information Science (JASIS), 46 (7), 478-489. Hersh, W., Pentecost, J. & Hickam, D. (1996) A task-oriented approach to information retrieval evaluation. Journal of the American Society for Information Science, 47 (1), 50-56. Hersh, W., Turpin, A., Price, S., Chan, B., Kramer, D., Sacherek, L. & Olson, D. (2000)."Do batch and user evaluations give the same results?" In: Proceedings of the annual international ACM SIGIR conference on Research and development in information retrieval pp.17-24. Athens, Greece: ACM. Hildreth, C. R. (2001) Accounting for users‟ inflated assessments of on-line catalogue search performance and usefulness: an experimental study. Information Research. Huffman, S. B. & Hochster, M. (2007)."How well does result relevance predict session satisfaction?" In: Proceedings of the annual international ACM SIGIR conference on Research and development in information retrieval pp.567-574. Amsterdam, The Netherlands: ACM. 16 Hufnagel, E. M. (1990)."User satisfaction-are we really measuring system effectiveness". In: Proceedings of the 23rd Annual Hawaii International Conference on System Science, pp.237-446. Hawaii, USA: IEEE Computer Society Järvelin, K. & Kekäläinen, J. (2000)."IR evaluation methods for retrieving highly relevant documents". In: Proceedings of the annual international ACM SIGIR conference on Research and development in information retrieval, pp.41 - 48 Athens, Greece: ACM. Johnson, F. C., Griffiths, J. R. & Hartley, R. J. (2003) Task dimensions of user evaluations of information retrieval systems. Information Research. Kokubu, T., Sakai, T., Saito, Y., Tsutsui, H., Manabe, T., Koyama, M. & Fujii, H. (2005)."The Relationship between Answer Ranking and User Satisfaction in a Question Answering System". In: Proceedings of the 5th NTCIR Workshop Meeting on Evaluation of Information Access Technologies (NTCIR-5), pp.537-544. Tokyo, Japan: Korfhage, R. R. (1997) Information Storage and Retrieval, New York, John Wiley & Sons. Lancaster, F. W. (1969) MEDLARS: report on the evaluation of its operating efficiency. American Documentation, 20 (2), 119-142. Lancaster, F. W. (1979) Information retrieval systems: characteristics, testing and evaluation, New York ; Chichester (etc.), John Wiley & Sons. Lancaster, F. W. (1981) Evaluation within the environment of an operating information service. IN SPÄRCK JONES, K. (Ed.) Information Retrieval Experiment. London, Butterworths. Lancaster, F. W., Cheryl, E., Mary, J. Z., Laura, M. & Yuen-Man, L. (1994) Searching Databases on CD-ROM: Comparison of the Results of End-User Searching with Results from Two Modes of Searching by Skilled Intermediaries. RQ, 33 (3), 370-386. Law, E. L.-C., Klobučar, T. & Pipan, M. (2006) User Effect in Evaluating Personalized Information Retrieval Systems. IN NEJDL, W. & TOCHTERMANN, K. (Eds.) LNCS 4227. Springer Berlin / Heidelberg. Marchionini, G. (1995) Information Seeking in Electronic Environments Cambridge University Press. Nielsen, J. (1993) Usability Engineering, San Francisco, Morgan Kaufmann. Sandore, B. (1990) Online Searching: What Measure Satisfaction? Library and Information Science Research, 12 (1), 33-54. Saracevic, T. & Kantor, P. (1988) A study of information seeking and retrieving. II. Users, questions, and effectiveness. Journal of the American Society for Information Science, 39 (3), 177-196. Soergel, D. (1976) Is user satisfaction a hobgoblin? ' Journal of the American Society for Information Science, 27 (4), 256-259. Spärck Jones, K. (1981) Information Retrieval Experiment, London, Butterworths. Steffey, R. J. & Meyer, N. (1989) Evaluating User Success and Satisfaction with CD-ROM. Laserdisk Professional, 2 (5), 35-45. Su, L. T. (1991). An investigation to find appropriate measures for evaluating interactive information retrieval. PhD, Rutgers University Su, L. T. (1992) Evaluation measures for interactive information retrieval. Information Processing and Management, 28 (4), 503-516. Su, L. T. (1998) Value of search results as a whole as the best single measure of information retrieval performance. Information Processing and Management, 34 (5), 557-579. Su, L. T. (2003) A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates. Journal of the American Society for Information Science and Technology, 54 (13), 1193-1223. Tessier, J. A., Crouch, W. W. & Atherton, P. (1977) New Measures of User Satisfaction With Computer Based Literature Searches. Special Libraries, 68 (11), 383-389. Thomas, P. & Hawking, D. (2006)."Evaluation by Comparing Result Sets in Context". In: Proceedings of the conference on information and knowledge management (CIKM), pp.94-101 Arlington, Virginia, USA: ACM. Turpin, A. H. & Hersh, W. (2001 )."Why batch and user evaluations do not give the same results". In: Proceedings of the annual international ACM SIGIR conference on Research and development in information retrieval, pp.225 - 231 New Orleans, Louisiana, United States ACM 17