8. Developing an evaluation plan
8.1 Getting started
8.2 Who has a stake in the evaluation: Stakeholder
analysis
8.3 Timeframe of concerns
8.4 Before moving on: Consider any constraints
8.5 Crystallising the evaluation questions
8.6 Choosing data gathering techniques
8.7 Storing the data
8.8 Articulation with SETL
8.9 Ethical considerations
8.1 Getting started
You may feel the task ahead a rather daunting one. However, good planning will pay off in terms of generating a plan that is manageable and likely to produce outcomes of worth and practical value.
To generate a good plan means logically working through a series of issues, and this section takes you through:
- consideration of stakeholders and their concerns
- consideration of constraints
- how to translate concerns into key evaluation questions
- selection of data gathering methods to address the key questions that are to be the focus of the evaluation.
When should planning begin?
Planning for evaluation should occur as part of the other planning activities associated with project start up. There are two good reasons for this:
- Aspects of the project will need evaluating during the formative stages of the project — for example its design, project processes, product prototype. Evaluation data will give the project team important feedback for refining/modifying development of the project outputs. Such data is an important part of quality assurance. Data therefore needs to be gathered soon after the commencement of the project, and ways of collecting that data have to be put in place.
- Considering what will be evaluated helps sharpen focus on the stated project goals and objectives. These need to be framed in a way that makes assessment of their achievement possible. The need for clearly stated 'output' and 'outcome' goals cannot be over-emphasised if evaluation is to be effective, as these serve as the key criteria, or standards, on which to base the evaluation. As far as possible and appropriate, these outcomes should be stated in measurable terms so that their realisation can be effectively evaluated (e.g. percent completion rates).
How to get planning underway
Once the planning/evaluation group is established and basic management issues addressed, we suggest you work through the following issues in a face-to-face workshop (assuming that a team approach to evaluation is envisaged). You will find a facilitator helpful — e.g. external consultant, Centre for the Advancement of Learning and Teaching (CALT) staff member, project leader. If scope for a face-to-face workshop is limited, consider videoconferencing or online activities (e.g. a discussion board).
8.2 Who has a stake in the evaluation: Stakeholder analysis
Stakeholder analysis involves:
- identifying the major stakeholders in your evaluation and,
- identifying the likely key concerns of those stakeholders.
Identifying stakeholders
Understanding the stakeholders and who the evaluation report(s) are actually for will shape the goals/objectives of the evaluation, the questions to be asked, when they are to be asked, and the methods of data collection, analysis and reporting - in other words, pretty well all facets of the evaluation!
Stakeholders could include:
- design/development staff
- teaching staff
- project manager/project steering group
- support staff
- Head of School, Dean of Faculty, Teaching & Learning Committee
students (users) - other groups — e.g. professional bodies, employers.
Identifying concerns
Stakeholders will likely differ in their concerns and what they want to find out. Column 3 of the following Stakeholder Analysis Worksheet [Word] outlines some of the typical concerns of stakeholders within the University. Note that the concerns are by no means mutually exclusive with respect to stakeholders, and the balance of concerns will vary from project to project. Importantly, concerns will change over the life of the project (from formative/development stage to implementation and beyond). Hence the questions asked will change over that time.
Note that this toolkit has a primary focus on learner (educational) processes and outcomes, and the balance of concerns reflects this bias.
| Activity 1 i. Add to the worksheet any further major stakeholders in your project, and their likely concerns. ii. Consult columns 1-3 of Table 2.1 (in Section 2.1) for a summary of the possible foci for educational evaluation and the reasons — or purposes — of focussing on these matters. Augment/adapt the concerns listed in the table above based on this analysis and your likely evaluation needs/concerns. iii. Critically check with the various stakeholders that you have indeed captured all their major concerns. |
NB: It is generally not feasible to address the concerns of all stakeholders (and indeed all the concerns of any one stakeholder) in any one evaluation exercise, so evaluators will generally need to limit the evaluation to what is practical and feasible. Time and costs will invariably constrain an evaluation program, and you will need to establish priorities.
| Activity 2 i. Prioritise the stakeholders and their concerns from 1 à X (with 1 of highest priority) in the two columns provided. (This should reflect who the evaluation is for.) ii. Refine if necessary the overall objectives for the evaluation to reflect these key concerns. |
8.3 Timeframe of concerns
Different stakeholders will want their concerns addressed and questions answered at different times, or phases, in the project–for example, during development, or following implementation (see Fig. 2.1 and Table 2.1 in Section 2).
Furthermore, certain questions can only be answered after suitable time has elapsed or phase in the project completed. For example, it may be some months after implementation before student performance data becomes available, and so questions on outcomes addressed. So both stakeholder needs and pragmatics will determine when particular concerns can be addressed.
Pre-implementation [Pre-imp]
Which concerns need to be addressed during the life of the project (up to the point of implementation) — i.e. during the formative phase of the project? (This might include design and development issues; project management issues.)
Post-implementation [Post-imp]
Which concerns need to be addressed following implementation of the project innovation:
a) |
in the short term [Post-imp.st] - e.g. first semester/year of implementation? (This might include gathering data regarding the perceived educational strengths and weaknesses/limitations of the innovation; data from the project team and teaching staff on management and implementation issues.) |
b) |
in the medium term [Post-imp.mt] — e.g. first/second year of implementation? (This might include analysis of student performance data and progress; uptake of the innovation within the school/faculty.) |
c) |
in the long term [Post-imp:lt] — e.g. two years plus? (This might include longitudinal and/or comparative studies; management issues related to mainstreaming of the innovation; cost-benefit analysis.) |
|
Activity 3 Identify each concern as either pre-imp, post-imp:st, post-imp:mt or post-imp:lt. |
8.4 Before moving on: Consider any constraints
Before finalising the questions that will direct the evaluation, consider any constraints to the evaluation that exist, or are likely to emerge as the evaluation unfolds. They might include:
- budget and resource constraints
- time constraints
- competent staff available to carry out the evaluation
- pre-specified evaluation objectives, methodologies and/or reporting procedures
- legal or ethical restrictions (see Section 8.9: Ethical considerations)
- availability of data, participants for information (e.g. if the evaluation is to occur over a semester break.)
- 'political' considerations
These factors will determine the size and scale of the evaluation and what the evaluation team can practically deal with.
| Activity 4 Jot down any constraints that will, or are likely to, limit your evaluation plan. |
8.5 Crystallising the evaluation questions
It's important to spend time in getting the evaluation questions right. Otherwise, you may get the wrong answers, or answers to questions you didn't ask or want to know about.
| Activity 5 i. Consider the concerns rated of highest priority from the previous activity. Rephrase each concern as a question — begin with terms such as 'What', 'How', 'When', 'Can', 'Will', 'Does', 'For whom', 'Under which circumstance's' etc. [See column 4 of Table 2.1 for typical questions based on the various educational foci identified.] ii. Which of the questions beg comparison (with other teaching strategies, student groups etc.) or measurement (test scores, percent retention etc.)? Rephrase these questions if necessary so that the parameters of measurement are clear. |
'Action' questions
One main purpose of evaluation is program improvement. Some questions deal with matters the project team can readily respond to, to rectify or improve an aspect of the innovation. Other questions may focus on more general or 'big picture' outcomes not as directly linked to action that can be taken by the project team or other key stakeholders — at least in the short term. It's therefore important to ask both specific, action-oriented as well as more general, 'big picture’ type questions.
Think through the possible answers to a particular question — do they give direct clues to changes that can be made? (i.e. Will the question actually inform change?)
'High value' questions
Some questions may be particularly useful to ask because of their high 'pay-off' because:
- there is little other information to inform in the area; hence answers will add real value to the information base
- the answers will be of great interest to the major stakeholders
- the answers will most likely significantly inform or highlight areas that can readily be improved
- the questions can be feasibly answered given the time and resources available.
One way to identify such 'high value' questions is to use a simple two-dimensional matrix:
- Dimension 1 relates to the value of the evaluation question in adding to the information base
- Dimension 2 relates to the value of the evaluation question in helping decision-making.
Questions in cell A are of highest value and should receive the bulk of your attention and resources; questions in cell D will probably not be dealt with unless they can be asked with minimal resources and time input.
[Adapted from Payne, 1994, pp. 48-49.]
| Activity 6 i. From the list of possible evaluation questions, select those that will be the foci for evaluation. ii. Record these questions on the Evaluation Plan template (see Section 12: Finalising the evaluation plan.) |
8.6 Choosing data gathering techniques
A general principle | Sources of data | A variety of methods | Factors to consider in selecting data gathering methods | Need for robustness and flexibility
The questions you propose to ask will determine the sorts of data you require (whether it be qualitative or quantitative data). In turn, there is a variety of techniques or approaches that can be used depending on the sorts of data that you want to gather.
Always keep in mind who wants the data, as different stakeholders will have preferences for the form of data put before them:
- Senior management and funding bodies may well prefer 'hard facts' and other quantitative data to show concrete outcomes for the money and resources invested.
- Developers may well prefer more qualitative data to inform improvements.
A general principle
It's best to use a number of data gathering techniques and/or sources of data to substantiate findings. This is known as a process of triangulation — the use of multiple investigative methods or information sources to home in on the question in focus.
Sources of data
Sources of data include:
- students — prospective, current, past, withdrawn
- colleagues — teaching partners, tutors, teachers external to the project
- discipline/instructional design experts
- professional development staff
- graduates and employers
- documents and records — teaching materials, assessment records, past SETLs, assessment statements and tasks
A variety of methods
The following tables, drawn from the Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education, outline suitable methods to use in the different phases of the project, and their particular purposes. As you can see from the web addresses cited, a valuable resource on methods is the Evaluation Cookbook produced by the Learning Technology Dissemination Initiative (LTDI) in the UK. The Cookbook not only tells you 'how to' but indicates the resources and indicative time needed to use each method. The web address is cited in following the tables. These tables in turn relate to Table 2.1: A learning-centred framework for whole project evaluation (see Section 2).
Table 8.1. Obtaining evidence for the Analysis and Design phase of the whole project evaluation framework
| Method and Purpose |
Further Information |
| Documentation To reveal the teacher’s or course designer’s assumptions and design decisions in structuring the project |
Teachers are asked to document and justify the decisions they have made at all stages of the design of their project. |
| Nominal group technique To identify key issues to be explored by other evaluation methods |
http://www.icbl.hw.ac.uk/ltdi/ |
| Analysis of Unit Materials To analyse the outcomes, objectives and assessment methods of the unit |
Table 8.2. Methods of obtaining evidence relevant to the formative evaluation of both the learning environment and the contextual learning processes
| Method and Purpose |
Further Information |
| Interviews and questionnaires To obtain student and peer comment on the attractiveness, usability and functionality of the innovation |
Interviews: Interface Questionnaire: Resource Questionnaire: Checklists: |
| Focus groups To elicit a range of student reactions to the innovation ret and prioritise the difficulties, or to interpret questionnaire responses |
http://www.icbl.hw.ac.uk/ltdi/ |
| Observation or video of students using the innovation To obtain a detailed understanding of the ways students use the innovation and the problems they encounter |
Observation: http://mime1.marc.gatech.edu/MM_Tools/ARF.html Video: |
| User tracking To obtain a detailed understanding of problems that students experience in using computer software, based on computer capture of the paths that students follow through the program. Requires specialised software |
http://www.icbl.hw.ac.uk/ltdi/ |
Table 8.3. Methods suitable for obtaining evidence relevant to the formative evaluation of the cognitive learning process
| Method and Purpose |
Further Information |
| Student ratings of learning confidence To judge how confident students are with the innovation's content |
http://www.icbl.hw.ac.uk/ltdi/ |
| Video of think aloud To record how students are thinking as they use the innovation |
Students are asked to verbalise what they are thinking as they use the innovation (such as computer software). In relation to software use, useful when thinking is not too demanding, but verbalising can ‘drop out’ under heavy cognitive loads. |
| Video-stimulated recall To reveal how students are thinking as they use the innovation |
Students are shown a video of themselves using the innovation and asked to say what they were thinking and why. Less prone to the ‘drop out’ problem, but reliant on the video to cue memories rather than confabulations. |
| Teach-back To reveal how a student’s understanding is linked to the innovation |
Students are asked to use the innovation to ‘teach’ the interviewer about the material, and in doing so to show how the innovation assisted their understanding. |
| Discussion archive To examine the nature of student discussion in ‘chat’ and discussion board (i.e. online) environments, or other environments |
Analysis of the interchanges between students in real time and asynchronous discussions, examining the nature of the interaction process and the quality of what is said. These may be compiled from electronic files or as transcriptions from audio tape etc. |
| Reflective journals To obtain students’ interpretations of the process of understanding and learning |
Students are asked to explain in writing how the innovation may have assisted them to develop their understanding and learning of key ideas, with emphasis upon the understanding and learning processes. Requires careful structuring and exemplification if the journal is to move beyond a fairly low-level description of events and experiences. |
Table 8.4. Methods suitable for obtaining evidence for summative evaluation of the learning process
| Method and Purpose |
Further Information |
| User tracking |
See Table 8.2 |
| Observation of students using the innovation |
See Table 8.2 |
| Video + think aloud |
See Table 8.3 |
| Video + stimulated recall |
See Table 8.3 |
| Teach-back |
See Table 8.3 |
| Reflective journals |
See Table 8.3 |
Table 8.5. Methods suitable for obtaining evidence for summative evaluation of learning outcomes
| Method and Purpose |
Further Information |
| Student confidence ratings To determine how confident students are with relevant areas of the unit |
http://www.icbl.hw.ac.uk/ltdi/ |
| Concept maps To reveal how students interrelate and characterise key concepts |
http://www.icbl.hw.ac.uk/ltdi/ |
| Clinical interview To reveal how a student thinks about an idea or principle and/or how s/he reasons or solves problems |
This method is often used in phenomenographic studies of students’ conceptions of key ideas–see for a detailed example. For general advice on interviewing: |
| Purpose-built assignments, exam questions To determine whether the innovation influences conventional learning outcomes |
As noted in Section 3: 'The place of assessment in evaluation' , standard assessments and grading procedures often are ill-suited to the evaluation of learning outcomes of new projects. Considerable care must be taken to ensure that the targeted learning is being tapped and graded appropriately. Assessment can take the form of pre- and post tests. |
| Focus groups |
See Table 8.2 |
Table 8.6. Methods suitable for obtaining evidence for summative evaluation of innovation appropriateness
| Method/ Documentation and Purpose |
Further Information/ Comment |
| Unit descriptions To record changes in curriculum emphasis |
Before and after comparisons of syllabus structures and assessments. Should be compared with students’ perceptions of emphases because of potential ‘hidden curriculum’ effects |
| Assessment records To look for changes in the patterning of achievement across different areas of the curriculum |
It may be difficult to document changes in students’ patterns of achievement if the assessments have been changed (from previous offerings of the unit) to optimise the fit with the innovation (see comments in relation to purpose built assessments in Table 8.5 above) |
| Student interviews To obtain students’ experiences of the curriculum, the emphases they adopted, and their reasons for doing so |
Individual: Group: |
| Student questionnaires To obtain evidence on, the emphases adopted by students |
General advice on questionnaire construction: |
| Peer and student ratings of pedagogical dimensions To localise aspects of the innovation that may not be experienced as intended |
Refer to articles by and . |
| Staff allocation records To note changes in patterns of staff support |
Before and after comparisons of staff deployment (quantum and pattern). |
Adapted from Phillips et al. (2000). Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education, pages 2.3, 2.4, 2.6, 2.7, as tables 2.2-2.7.
For further information on these and other methods
References cited in the tables
Hargreaves, M. H. (1999). Evaluation of technological assisted learning. Paper presented at the 16th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE) 1999), Brisbane.
Lybeck, L., Marton, F., Stromdahl, H., & Tullberg, A. (1988). The phenomenography of the ‘mole concept’ in chemistry. In P. Ramsden (Ed.), Improving learning: New perspectives (pp. 81-108). London: Kogan Page.
Reeves, T. C., & Laffey, J. M. (1999). Design, assessment and evaluation of a problem-based learning environment in undergraduate engineering. Higher Education Research and Development, 18(2), 219-232.
http://www.icbl.hw.ac.uk/ltdi/cookbook/ The Evaluation Cookbook produced by the Learning Technology Dissemination Initiative at Heriot-Watt University, Edinburgh. Full of practical 'recipes' as well as practical advice on planning and preparation and reporting. See links in tables above.
http://mime1.marc.gatech.edu/MM_Tools/evaluation.html Web site provided by Georgia Tech Research Institute. Provides templates for gathering a range of qualitative and quantitative data using a variety of evaluation methods.
Other web sites
http://iet.open.ac.uk/plum/evaluation/plum.html Web site produced by the Institute for Educational Technology and the Open University, UK. Has information on data collection methods, and includes a number of pro-formas or templates for recording data.
Checklists to assist developers evaluate computer-based courseware
See Appendix to this kit. Four different checklists are cited and/or provided:
- A ‘ratings’ checklist to evaluate courseware during development, or a commercial product (Multimedia/online unit quality checklist [Word])
- University of Alberta checklist of ‘Principles for Effective Instructional Multimedia’
- A series of ‘heuristic’ tables to evaluate interface and education design, and content — from Peter Albion, University of southern Queensland.
- Learning
design evaluation (pdf)- An evaluation worksheet based on principles
of high quality learning activities. From an AUTC grant through the
University of Wollongong.
Factors to consider in selecting data gathering methods
Obviously, 'fitness for purpose' is the key factor here — will the particular technique(s) yield the data you want, when you want it, and help answer the particular question you have?
Other factors to consider are:
- the particular paradigm for the study (empirical, interpretive, critical theory-based, pragmatic)
- the time involved in preparing to use the particular method/tool (e.g. preparation of a bank of questions for a questionnaire)
- the time involved in gathering or recording the data — on the part of the data collector; on the part of the 'evaluee/s'
- the time needed to analyse and report the data (viz-a-viz when the findings are required)
- the scale involved — the number of students, staff required for valid/authentic data.
- the costs involved — in collection, analysis, and reporting. (See Section 10 for elaboration of possible costs here.)
These
factors will determine the balance/mix of quantitative and qualitative
data gathering techniques. It will also shape the range of data available
for analysis, as different techniques circumscribe to varying degrees
the sorts of data that can be collected. For example, a Likert scale type
questionnaire limits the data to be gathered; data from an unstructured
interview or student diary can be far less constrained.
Then you need to consider:
- the skill/expertise required to use the method
- the expertise, personnel and/or resources required to analyse and/or report the data.
|
In summary, your evaluation design needs to be feasible in terms of budget, schedule, personnel availability and data availability — i.e. it must be realistic. |
Data gathering and processing costs are but one budget item to consider. See Section 6: Costing an evaluation for other costs.
Need for robustness and flexibility
Events can conspire against the evaluation team! For example, a data source may drop out. Is your data collection plan able to withstand such a loss? Do you have back up methods to target the same sort of data (noting the recommendation regarding triangulation earlier in Section 8.6)?
Other unforseen events may occur, so your design needs to be reasonably flexible. Evaluation designs usually evolve in some way, as the evaluators interact with the various stakeholders during the course of the evaluation. For example, data obtained may indicate that a question needs further exploration or rephrasing. Other 'gaps' in intelligence may become evident. A robust design will allow for this growth and change, yet preserve the overall intent.
Using a matrix to help select suitable data gathering methods
|
Activity 7 Use the Evaluation Matrix Worksheet [Word] as a prompt to consider the range of data gathering options available, and to identify those most appropriate and feasible for each particular evaluation question.
|
8.7 Storing the data
Finally, you need to consider how and where you will store the data to aid retrieval and the analysis that will follow. This means:
- making sure that data is safe and not lost
- thinking through filing categories; e.g. by question type; data source; data method
- considering confidentiality requirements and any other safeguards, as well as authorisation arrangements to access data, e.g. password protected files, locked filing cabinet.
8.8 Articulation of evaluation with SETL
Depending on the project, the evaluation team may wish to incorporate a formal University of Tasmania SETL (Student Evaluation of Teaching & Learning) survey as part of the overall summative evaluation program. The team should familiarise themselves with SETL procedures regarding the collection, storage and analysis of data, and the distribution of findings. See http://student.admin.utas.edu.au/setl/index.html
8.9 Ethical considerations
In any evaluation, the rights and welfare of 'subjects' (staff, students etc.) need to be respected and protected. Two considerations stand out:
1. |
Privacy: Some data gathering techniques may be perceived as an invasion of privacy if prior consent on the part of the subject(s) has not been gained. |
2. |
Confidentiality: Much information that subjects provide is given in confidence unless specific permission to use 'private' information (such as names) has been given. Procedures relating to the collection, storage and retrieval of data must take the maintenance of confidentiality into account. |
A third issue worth consideration relates to the proposed use of 'test' and 'control' groups in implementing an innovation. Should an innovation of expected benefit be withheld from a control group?
In all situations, it is important to make clear how you intend to use the data, and to seek participants' agreement.
You should be aware of the University’s policies and guidelines regarding ethics approval for human subjects. Such approval, if required, should be obtained at the beginning of the project, and varied if required. Contact the Research Office or visit their site at http://www.research.utas.edu.au/ if you require clarification on issues around human subjects.