2. A project evaluation framework
2.1 Project evaluation
2.2 A focus on educational evaluation
2.3 Other foci
2.1 Project evaluation
By definition, a project has:
- a clearly defined beginning and end, with identifiable stages (or phases) between; and
- specified outputs and outcomes that are reflected in the stated goals/aims and objectives of the project.
In a nutshell, evaluation is about making judgements of ‘worth’. In the context of educational projects then, this means judgements about how well the project’s objectives have been achieved (in terms of outputs and educational outcomes), and how well the project has run as an enterprise.
These facets, or aspects, of a project can serve as the key foci or framework for evaluation — see Figure 2.1 below.
Figure 2.1 Project evaluation
framework
Evaluation beyond the life of the project
Whilst projects, by definition, have an ending and defined outputs, the outcomes may take time to manifest themselves (e.g. as improved performance on assessment tasks, better retention rates etc.) Furthermore, educational projects may not cease upon production of the 'outputs' — they may not end with a neat 'handover' and sign off (as in say the commissioning of a building) but rather continue in some form, over a number of semesters. For both these reasons, evaluation will continue into implementation and beyond.
2.2 A focus on educational evaluation
Teaching and learning projects are ultimately primarily concerned with student outcomes; i.e. on educational outcomes, and the bulk of this toolkit is thus to do with educational evaluation.An integrated framework for educational evaluation is shown in Table 2.1. (The table is in A3 Word format.) The text of the table is in HTML format at the end of this section.
Features of the integrated educational project evaluation framework
Four phases are evident in an educational project:
- Each phase provides a focus or number of foci for evaluation. It’s important to include each phase in the overall evaluation of the project.
- There is a clear purpose, and set of evaluation questions for each focal point. The questions are student/learner-centred.
- The questions asked shape the type of evidence, and evidence gathering techniques.
Table 2.1: A learning-centred framework for whole project evaluation [Adapted from and Phillips et al (2000)]
Summative Formative |
|||
Phase 1: Analysis and Design |
|||
| Focus |
Purpose |
Questions |
Evidence and methods |
| 1.1 Curriculum analysis |
To describe the inadequacies/ insufficiencies of the current curriculum, with particular attention to the shortfall in student learning. |
What is the learning need and why can’t it be met with existing teaching/ learning arrangements? The curriculum analysis should result in a clear statement of the desired learning outcomes–i.e., the learning objectives. |
Analysis of the nature of the shortfall in student learning and the probable reasons for it. The analysis is conducted through self-, peer and expert review, review of documentation, informed by relevant educational literature, with attention to the interplay between content, teaching/learning activities and assessments. |
| 1.2 Teaching-for-learning analysis |
To describe and justify the teaching/ learning/ assessment process likely to bring about the desired learning outcome. |
What teaching/ learning/ assessment process is likely to meet the learning need? The teaching-for-learning analysis should be informed by the literature. |
Description of the proposed teaching/learning process with argument indicating why it is likely to redress the shortfall in learning outcome, based on the evidence from the literature. |
| 1.3 Specification of innovation |
To describe and justify the proposed implementation, and indicate how it will facilitate the desired learning process and outcome (in 1.2) |
How does the innovation actualise the desired teaching/learning/ assessment process? |
Educational plausibility is established by detailing the implementation within the course context, and specifying how the learning process and outcome will be assessed. Project feasibility is judged through demonstration of proof of concept (e.g. through prototyping, success in a similar context) plus peer/expert review of anticipated costs and institutional 'fit'. |
| Phase 2. Development |
|||
| Focus |
Purpose |
Questions |
Evidence and methods |
| 2.1 Formative monitoring of learning environment |
To determine whether the innovation is functional in its context and accessible/ attractive to students (and modify as needed). |
Can students use the innovation easily (e.g. can they navigate, gain access to materials, perform the requisite tasks, etc) and do they find the innovation attractive, approachable and accessible? |
Evidence focused on the workability of the innovation and student involvement with it: observation; video; user tracking — if computer-based; student and peer reactions (expressed through interviews, focus groups, questionnaires). Viability modifications determined through peer and expert review. |
| 2.2 Formative monitoring of learning process |
To determine whether the innovation is influencing the learning process as intended (and modify as needed). |
Does the manner in which students use the innovation encourage the desired learning process and is there evidence that the desired learning is occurring?
|
Evidence focused on the nature of the learning process and its immediate consequences: video; think aloud; stimulated recall; teach-back; reflective journals. Viability/modifications determined through peer and expert review |
| Phase 3. Implementation |
|||
| Focus |
Purpose |
Questions |
Evidence and methods |
| 3.1 Summative evaluation of learning process |
To determine whether the innovation is influencing the learning process as intended. |
Now that the innovation is part of the unit and not ‘experimental’, do students use it in the way intended?
|
Evidence focused on the nature of the learning process viz-a-viz that intended; methods as per 2.2 above, plus user tracking. |
| 3.2 Summative evaluation of learning outcome |
To determine whether the learning outcome is as intended. |
The questions under this heading shift from a focus on what is happening as the innovation is being used, to its impact on achievement towards the completion of the unit. For example:
|
Evidence focused on the nature of the learning outcome using outcome-relevant assessment tasks, supported by conventional assessments and student interviews and questionnaires where appropriate. |
| 3.3 Summative evaluation of innovation appropriateness |
To determine whether the innovation is educationally appropriate in its immediate context. |
|
Peer and expert review of the educational worth and viability of the innovation in the unit/subject concerned, based on evidence gathered in 2.1, 2.2 and 3.2, plus evidence on integration of the innovation into the curriculum. |
| Phase 4. Institutionalisation |
|||
| Focus |
Purpose |
Questions |
Evidence and methods |
| 4.1 Impact evaluation |
To determine the robustness of the learning and its transfer beyond the immediate context of the innovation. |
|
Evidence of beneficial effects on: understanding and learning in related/subsequent areas of the curriculum; indirect indicators (e.g. progress and retention rates); development of generic capabilities; transfer to the workplace. |
| 4.2 Maintenance evaluation |
To determine the sustainability of the innovation in the context of the whole course. |
|
Peer and expert review of the educational benefits of the innovation considered in relation to its maintenance and opportunity costs, and in relation to the educational and funding policies of the institution. |
2.3 Other foci
Project processes
Although the main focus of this toolkit is on learner-centred evaluation, Section 9 of the kit deals with evaluation of project processes — planning, communications, management etc. in Figure 2.1. See Section 9: Evaluating project processes.
Program evaluation
For more general project or program evaluation approaches and methodologies see, for example:
Brinkerhoff, R. O., Brethower, D. M., Hluchyj, T. and Nowakowski, J.
R. (1983). Program evaluation : a practitioner's guide for trainers
and educators :
Boston : Kluwer-Nijhoff.
Owen, J. M. (1999). Program evaluation: forms and approaches. 2nd ed.
St Leonards, N.S.W. : Allen & Unwin.
Payne, D. A. (1994). Designing educational project and program evaluations. Boston: Kluwer Academic Publishers.
Worthen, B. R., Sanders, J. R. and Fitzpatrick, J. L. (1997). Program evaluation: alternative approaches and practical guidelines. 2nd ed. New York : Longman.
References
Alexander, S., & Hedberg, J. G. (1994). Evaluating technology-based learning: which model? In K. Beattie, C. McNaught, & S. Wills (Eds.), Multimedia in Higher Education: Designing for Change in Teaching and Learning . Amsterdam: Elsevier.
Bain, J. (1999). Introduction to 'Learning-centred evaluation of innovation in higher education'. Higher Education Research & Development 18 (2), pp. 165-172.
Phillips, R., Bain, J., McNaught, C., Rice, M. and Tripp, D. (2000). Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education, page 1.5.