Evaluation of Environmental Decision and Information Support Tools: from Adoption to Outcome
19th International Congress on Modelling and Simulation, Perth, Australia, 12–16 December 2011 http://mssanz.org.au/modsim2011
Evaluation of environmental decision and information support tools: from adoption to outcome
B.S. McIntosh a, b b International WaterCentre, Brisbane, Australia Smart Water Research Centre, Gold Coast, Australia Email: b.mcintosh@watercentre.org
a
Abstract: The value of environmental decision and information support tools (DISTs) and technologies is located in the impacts that the use of such technologies have on the behaviour of individuals and organisations, and consequently on our collective ability to negotiate the difficult interfaces between human social and economic activity, and environmental impact. There are significant, but as yet insufficiently exploited opportunities for improving the value of these tools and technologies through the use of assessment and evaluation practices. Iterative and evolutionary tool development processes offer the opportunity to deliberately incorporate user assessment information into design. Doing so can improve adoption and use impacts through better tailoring of tool functionality, output and interfaces to user needs. Managing such processes successfully is demanding and we do not possess sufficient, generalised or published practical understanding of what works and what does not in terms of user interaction. However, there is an emerging understanding of how environmental DIST teams should be structured and a wide body of knowledge yet to be imported from the software, product and interaction design communities, so the future is promising. The evaluation of the impact of using environmental DISTs is more problematic and less well understood, both methodologically and empirically. Our conceptual frameworks for characterising use and identifying where impacts arising from tool use might lie are nascent and require refinement, both to reflect the variety of developmental routes and
Evaluation of environmental decision and information support tools: from adoption to outcome
B.S. McIntosh a, b b International WaterCentre, Brisbane, Australia Smart Water Research Centre, Gold Coast, Australia Email: b.mcintosh@watercentre.org
a
Abstract: The value of environmental decision and information support tools (DISTs) and technologies is located in the impacts that the use of such technologies have on the behaviour of individuals and organisations, and consequently on our collective ability to negotiate the difficult interfaces between human social and economic activity, and environmental impact. There are significant, but as yet insufficiently exploited opportunities for improving the value of these tools and technologies through the use of assessment and evaluation practices. Iterative and evolutionary tool development processes offer the opportunity to deliberately incorporate user assessment information into design. Doing so can improve adoption and use impacts through better tailoring of tool functionality, output and interfaces to user needs. Managing such processes successfully is demanding and we do not possess sufficient, generalised or published practical understanding of what works and what does not in terms of user interaction. However, there is an emerging understanding of how environmental DIST teams should be structured and a wide body of knowledge yet to be imported from the software, product and interaction design communities, so the future is promising. The evaluation of the impact of using environmental DISTs is more problematic and less well understood, both methodologically and empirically. Our conceptual frameworks for characterising use and identifying where impacts arising from tool use might lie are nascent and require refinement, both to reflect the variety of developmental routes and
References: Checkland, P., and Holwell, S. (1998). Information, Systems, and Information Systems: Making Sense of the Field. John Wiley and Sons, Chichester. Checkland, P., and Scholes, J. (1990), Soft Systems Methodology in Action. John Wiley and Sons, Chichester. Cooper, A. Reimann, R., and Cronin, D. (2007). About Face 3 : The Essentials of Interaction Design. John Wiley and Sons Inc., Chichester. Diez, E., and McIntosh, B.S. (2009). A review of the factors which influence the use and usefulness of information systems. Environmental Modelling and Software 24:588-602. Diez, E., and McIntosh, B.S. (2011). Organisational drivers for, constraints on and impacts of decision and information support tool use in desertification policy and management. Environmental Modelling and Software 26:317-327. Holtzblatt, K., Burns Wendell, J., and Wood, S. (2005). Rapid Contextual Design, A How-To Guide to Key Techniques for User-Centred Design. Morgan Kaufman, San Francisco. Matthews, K.B., Rivington, M., Blackstock, K., McGrum, G., Buchan, K., and Miller, D.G. (2011). Raising the bar? – The challenges of evaluating the outcomes of environmental modelling and software. Environmental Modelling and Software 26:247-257. McCown, R.L. (2002). Changing systems for supporting farmers’ decision making: problems, paradigms and prospects. Agricultural Systems 74:179-220. McIntosh, B.S., Ascough II, J.C., Twery, M., Chew, J., Elmahdi, A., Haase, D., Harou, J., Hepting, D., Cuddy, S., Jakeman, A.J., Chen, S., Kassahun, A., Lautenbach, S., Matthews, K., Merritt, W., Quinn, N.W.T., Rodriguez-Rodan, I., Sieber, S., Stavenga, M., Sulis, A., Ticehurst, J., Volk, M., Wrobel, M., van Delden, H., and El-Sawah, S. (accepted), Environmental Decision Support Systems (EDSS) development – challenges and best practices. Environmental Modelling and Software Nilsson, M., Jordan, A., Turnpenny, J., Hertin, J., Nykvist, B., and Russel, D. (2008). The use and non-use of policy appraisal tools in public policy making: an analysis of three European countries and the European Union. Policy Science 41:335-355. Oxley, T., McIntosh, B.S., Winder, N., Mulligan, M., Engelen, G. (2004). Integrated modelling and decisionsupport tools: a Mediterranean example. Environmental Modelling & Software 19 (11) 120-125. Reeve, D., and Petch, J. (1999). GIS Organisations and People, A Socio-Technical Approach. Taylor and Francis, London. Rouwette, E.A.J.A., Vennix, J.A.M., and van Mullekom, T. (2002). Group model building effectiveness: a review of assessment studies. System Dynamics Review 18:5-45. Tullis, T., and Albert, B. (2008). Measuring the User Experience: Collecting, Analyzing and Presenting Usability Metrics. Morgan Kaufmann, San Francisco. United Nations (2010). World Population Prospects, The 2010 Revision. http://esa.un.org/unpd/wpp/ van Delden, H., Seppelt, R., White, R., and Jakeman, T. (2011). A methodology for the design and development of integrated models for policy support. Environmental Modelling and Software 26: 266-279. Voinov, A. and Bousquet, F. (2010). Modelling with stakeholders. Environmental Modelling and Software 25:1268-1281. 103