Download this section

1. Introducing the Project Evaluation Toolkit

1.1 What the toolkit is for
1.2 Use of the toolkit
1.3 Recognising evaluation perspectives

1.1 What the toolkit is for

Evaluation is the cornerstone for program improvement, but that evaluation must be carefully planned for it to be effective. This toolkit takes you through the planning and development phase for an evaluation program targeted at an educational innovation/project at the University of Tasmania. It then provides advice and links to resources that help you and/or the evaluation team to carry out the evaluation — from data collection, analysis, to reporting the findings.

The nature of University projects varies considerably — from initiatives in online/technology-based course delivery to restructuring of whole programs to better meet the needs of students and/or realise faculty/school imperatives. This very diversity of projects means that no one evaluation methodology or one suite of techniques will ‘fit’ all — rather, there is a need to match the method(s) with the particular program that is the focus of evaluation. In addition, the needs, or reasons, for an evaluation may vary; there are invariably a number of different stakeholder groups involved in any project — from developers to funding agents, as well as students — and this influences the questions to be explored, which in turn influences the evaluation methodologies and techniques to be used.

So projects are diverse, and the needs/reasons for evaluation vary according to stakeholders. Needs also vary over time. Foci for evaluation during the development phase of a project will differ from those following implementation, for example.

This toolkit presents a range of evaluation methodologies and specific techniques, and facilitates the decision-making process in deriving an evaluation plan appropriate to the project and needs of the evaluation. The kit is based on an integrated, learning-centred project evaluation framework (see Fig 2.1: Project Evaluation Framework and Table 2.1: A learning-centred framework).

Focus

Its primary focus is on the evaluation of student learning — hence the majority of questions and strategies focus on this aspect of evaluation.

A secondary focus is on the impact of the project on staff and faculty/school practices and attitudes.

1.2 Use of the toolkit

The toolkit is primarily for teaching and development staff engaged in teaching/learning projects within the University. It makes no assumptions about prior knowledge of evaluation. For a general introduction to evaluation, see http://www.utas.edu.au/tl/improving/evaluation/index.html

It can also be of value to evaluators/consultants called in to assist with the evaluation of projects.

The toolkit can be used in either of two ways:

You can tackle each item of the toolkit in sequence, building up your evaluation plans as you go. There is a variety of activities and worksheets to assist you in preparing the plan, culminating in a completed design map (see Section 11: Finalising the evaluation plan).
You can simply dip into sections of interest to you; for example to find out about the range of data gathering techniques possible (see 6.6).

1.3 Recognising evaluation perspectives

Evaluators differ in the perspectives they bring to the task — this influences both the questions they ask and the strategies they use. In turn, outcomes of an evaluation are shaped to an extent by the design of the evaluation. It is therefore important that evaluators recognise the evaluation perspective (or 'paradigm') they bring to, and adopt, in any evaluation. Reeves (1997) identified the dominant paradigms evident in evaluation studies, and these are summarised by Phillips et al. (2000) in the following table [table adapted].

Table 1.1: Brief summary of the dominant evaluation paradigms

Paradigm	Assumptions	Comment
Positivist-Quantitative Paradigm	Problems can be defined a priori. The complexity of social situations can be reduced to a string of variables which are clearly operationalised. There is a reliance on controlled experimentation. Events can be explained in terms of cause and effect. There is one ‘right’ interpretation.	Represents the traditional empirical approaches to evaluation. There can be value in seeking to quantify measures. However, people and the complexity of social interactions cannot be reduced to clearly defined variables, and it often is impossible to produce matched groups of people for comparative studies. We would advise users against adopting only quantitative strategies.
Constructivist-Interpretive -Qualitative Paradigm	There is a focus on exploring the dynamics of interactions with the emphasis on the world as a socially constructed reality involving multiple perspectives. The perceptions and values of all the participants in a situation are needed in order to explore the various possible interpretations.	This paradigm has enriched our understanding of social situations a great deal. The main problem with the qualitative nature of this approach is that it does not necessarily focus on the areas which need change. Descriptions are made, but often without any form of judgment attached. This is at odds with the attempt to find appropriate ways to improve situations, which may be the purpose of the evaluation.
Critical Theory- Postmodern- Paradigm	Critical theory aims to transcend the positivism of the traditional approach and the relativism of the interpretive approach by placing the process of critical reflection at the centre of the research process. The focus is on changing the world, not only describing it. The concept of praxis is important; praxis is action which is informed by theoretical ideas and by the process of reflection on existing practice. Theory and reflection feed into the formulation of new practice.	Action inquiry has strong links to critical theory. In both a Constructivist-Interpretive -Qualitative approach and Critical Theory- Postmodern approach, understanding the dynamics and multiple perspectives of those involved is important. Qualitative strategies are used in both, but the distinction lies in the purpose to which the evaluation will be put.
Eclectic-Mixed Methods-Pragmatic Paradigm	This approach is more capable of handling the complexity of modern society and technology. The focus is on practical problems rather than on issues of reality and theories of society. It acknowledges the weakness of current evaluation tools.	Complex evaluation plans can result. The strength of this approach is the acknowledgment of the current state of the art of evaluation; there are no ‘right’ approaches and maintaining an open approach is essential. We recommend that users look favourably at this pragmatic way of proceeding.

Phillips et al. (2000). Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education, page 1.5.

This toolkit is based on the Eclectic-Mixed Methods-Pragmatic Paradigm — it provides a variety of qualitative and quantitative strategies/techniques and approaches (yielding a variety of quantitative and qualitative data), and users are led through a series of steps in order to select those strategies that best meet their evaluation needs.

References

Reeves, T. C. (1997). Established and emerging evaluation paradigms for instructional design. In C. R. Dills & A. J. Romiszowski (Eds.), Instructional Development Paradigms (pp. 163-178). Englewood Cliffs, New Jersey: Educational Technology Publications.

Phillips, R., Bain, J., McNaught, C., Rice, M. and Tripp, D. (2000). Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education, Murdoch University and the Australian Society for Computers in Learning in Tertiary Education (ASCILITE). Available http://www.tlc.murdoch.edu.au/archive/cutsd99/handbook/handbook.html

Authorised by the Director, Centre for the Advancement of Learning and Teaching (CALT)
© University of Tasmania
Copyright and Disclaimers | Accessibility | Feedback, Suggestions and Questions

Last updated: 21-Nov-2005