7. Organising for an evaluation

7.1 Introduction
7.2 Who should be involved in organising an evaluation?
7.3 Who should carry out the evaluation?
7.4 Selecting an external evaluator
7.5 Establishing policies, procedures and structures

7.1 Introduction

Organising for a project evaluation will involve:

• putting together an evaluation team
• developing an evaluation plan, and
• putting in place a management structure and process so that the evaluation enterprise runs smoothly.

In fact, the evaluation is a project in itself, requiring planning, scoping, and management of personnel and resources.

This section deals with setting up the evaluation team and establishing a management framework within which planning for the evaluation can begin.

7.2 Who should be involved in organising an evaluation?

Ideally, all major 'stakeholders' (i.e. those who have an investment in the outcomes of the project) should be involved — particularly in framing the evaluation objectives and evaluation questions. This helps create a sense of ownership and makes more likely that the findings will be acted upon. Certainly, those who you will expect to implement the recommendations of the evaluation ought be involved in the planning of the evaluation. Stakeholders can include design/development staff, teaching staff, students and funding agencies. (See Section 8.2: Who has a stake in the evaluation: stakeholder analysis.)

7.3 Who should carry out the evaluation?

For example, should it be:

• one individual, or a number of people?
• those inside or external to the project/program, or both?

Factors to consider include:

The degree of independence or 'objectivity' required:
This relates to whom the evaluation is for. It may be by and on behalf of the teaching staff or development team, as a piece of action research to better inform practice — here an evaluation team of the 'practitioners' is appropriate. Alternatively, the evaluation may be for an external funding agency where a more 'independent' analysis may be called for.

Time, cost and logistics:

• Is there a budget to employ an external evaluator?
• Do internal project staff have the time, or are they able to get time release?
• Is a team feasible logistically, or is it more expedient to have one person act as evaluator, with overall responsibility for planning and carrying out the evaluation, in close collaboration with developers, designers and users (teaching staff and students)?

  [An alternate model may be where the external expert acts as an evaluation facilitator for the project/program team, assisting the team in the planning and carrying out of the evaluation. The external evaluator may carry out the bulk of the data gathering and analysis.]

Costs will be something that must be kept in mind throughout the evaluation planning process. Section 6: Costing an evaluation lists typical cost items.

Expertise:

• How confident is the project team in being able to use the data gathering, analysis and reporting techniques envisaged?
• What level (if any) of outside assistance is needed?

An external professional evaluator may be able to develop or apply a method more efficiently; however, an 'in-house' project team better understands the context, the nature of the innovation, and has better access to data, and because of this may in fact be able to evaluate more quickly than the external professional.

A general recommendation if an external party is involved

Split or allocate evaluation responsibilities between the internal team and the external partner(s), with the project team largely responsible for formative evaluation and the external party largely for summative evaluation. Then work closely and cooperatively together. The intimate knowledge of the project team can blend well with the expertise of the external evaluator.

7.4 Selecting an external evaluator

Those involved in an evaluation — whether they be internal or external evaluators — have a variety of tasks and responsibilities. Some will be shared, others will be the responsibility of a particular person. To carry out these responsibilities requires a range of competencies. According to Payne (1994) the competent evaluator should be able to:

• 'Specify information needs for program planning and evaluation.
• Develop a plan or evaluating specified questions.
• Locate, read, and integrate relevant research, measurement, and evaluation literature.
• Specify evaluation objectives and data base requirements in appropriate form(s).
• Critically evaluate a given evaluative research design.
• Relate theoretical evaluation models and real-life requirements.
• Relate input, transaction, and outcome variables.
• Demonstrate appropriate interpersonal relationship skills in working with evaluation team and program staff.
• Differentiate advantages and disadvantages of cross- sectional and longitudinal studies.
• Conduct systems, functions and task analyses.
• Design an effective measurement-management system.
• Describe evaluation design and analysis requirements in computer programmer or data-processing terms.
• Specify criteria for selection or development of evaluation instruments.
• Apply appropriate data-gathering procedures.
• Make a cost-benefit analysis of a given program or project.
• Use evaluation information to make decisions about programs or projects.
• Administer the activities of the evaluation unit.
• Design a system of data presentation that describes format, responsibility, procedures, recipients, and schedule.
• Redesign and refine evaluation systems based on data implications of previous cycle. Create reports that communicate.'

[Adapted from Payne, D. A. (1994). Designing Educational Project and Program Evaluations: A Practical Overview Based on Research and Experience. Kluwer Academic Publishers. Boston. (pp. 176-77]

 

7.5 Establishing policies, procedures and structures

Before the actual planning of the evaluation can get underway, some fundamentals need to be established. Answers to the following questions are needed:

Here is an Evaluation Management Framework Worksheet [Word] to record your answers to these questions.