New Multi- Evaluation options within Assignments

Note: This product is in Beta and the University of Tasmania has been given early access to use and test the functionality.

Overview:

There are now additional advanced options to manage who evaluates a student submission.

Assignment Creation:

When creating a new assignment or editing an existing assignment, in the side panel under Evaluation & Feedback, we display a count of evaluators who can give feedback in the unit. When you click on “Edit Evaluators” an Evaluators list dialog displays all evaluators with "Give Feedback" role permission in the unit.

About Evaluators:

You can assign evaluators to the assignment by selecting evaluators in this list. If you do not make any selections, the advanced assessment capabilities will not be available. However, all evaluators who have permission to give feedback will have access to this activity. Selecting the Manage Evaluators option opens the evaluator selection window to allow you to make changes to the evaluators for this assignment.

After the evaluators are assigned, we reveal three new capabilities in the side panel: (1) Evaluators, (2) Publishers and (3) Advanced Evaluation. NOTE: When enrolment changes are made for evaluators in a unit, it could take up to two minutes for this enrolment change to be processed and reflected in the Edit Evaluators list and in the Evaluator mapping table.

About Publishers:

While creating or editing an assignment with assigned evaluators, you can assign publishers based on the three options listed below. Depending on the option selected, a publish button will appear on the evaluation screen for users who meet the criteria of that option.
Note that only publishers can revert evaluations already released to students.
1. All evaluators can publish any evaluations – This means that anyone assigned as an evaluator can also publish the evaluation.
2. Evaluators can only publish their evaluation – This means that only the evaluator assigned to evaluate a specific student submission can publish the evaluation. You can review the Advanced Evaluation dialog to learn which evaluator gets to publish which student submission.
3. Manually select publishers – This means you can select one or more users from a list of users who have “Give feedback” permission to be the publisher. A list of users with “Give feedback” permission show up, and then a selection can be made to decide who publishes the evaluation to the learner.

After selection the publication method, a window displays all the available publishers in the unit. One or more publishers can be selected from this list. Select the Save button to save the selection.

About Allocating Students to Evaluators

Selecting the Manage Matching link will open the Student to evaluator mapping window. Options are available in the drop-down list to determine how the students should be matched to the evaluators. The system can randomly match each student to one, two or three evaluators. However, it is also possible to customize the evaluator and student mapping to choose more specifically who evaluates who using the student-evaluator mapping table and selecting or un-selecting the matching checkboxes.

Two options are available for evaluation modes which allow you to assign evaluators quickly.
(1) Single shared evaluation. All evaluators contribute to the same shared evaluation of the student submitted work . This is often referred to as “co -marking”.
(2) Multiple separate evaluation. Each evaluator will complete his or her own individual evaluation separately. Those independent evaluations will then be aggregated by the publisher and released to the student as a composite grade. This is often referred to as “independent multi-marker”.

If a new student enrolls in the unit after the mapping has been done the system will allocate the new user automatically based on the allocation options for the assignment. It is also possible to clear all the mappings in the table at once using the Clear Table button. Evaluator mapping can also be assigned by groups, this is done by using the groups drop down selector. This group will then display only the students to be mapped.

About Quick Eval:

When the advanced options are set for the assignment, then in the Quick Eval Submissions view the evaluator will only see the student submissions for the assignments they have been assigned to evaluate.

If the user is a publisher, then the publisher will see all submissions.

Assignment Evaluation:

On the Assignment List Page, the evaluators will see submissions all learners in the unit. However, the evaluator can evaluate only students they are assigned and not everyone in the unit. The evaluator will see an “Evaluate” button. If the user is not assigned to evaluate the student, then they see a “View Submission” button. If multiple evaluators have been mapped for the assignment and where that mapping does not have evaluator assessing each student, a “Pending Evaluation” link will appear for the students which are not mapped to that evaluator.

On the evaluation page, the assigned evaluator(s) will be displayed. If the assignment has been created with multiple evaluators, each evaluator will see a separate evaluation tab in the right panel for their evaluation. If the assignment has been configured for multiple evaluations, then the right panel will contain fields for an evaluators specific rubric evaluation and feedback. Each evaluator can see the other evaluators evaluations and feedback, but cannot edit or save on any tab other than their own.

When the evaluation is completed, the evaluator should select "Ready to Aggregate" button.

For the publisher, when the individual evaluators have completed their evaluation, checkmarks will appear in the evaluation view. The publisher will then be taken to the aggregated evaluation tab. The publisher can see both of the individual evaluations but cannot edit them or make any changes. The publisher will complete the aggregated evaluation and then select the Publish button (or save as draft and publish all at a later time).

FAQ and Known Limitations:

1. Assignment Copy - The options and student-evaluator mapping persist upon copying an assignment within the same unit. However, when copied to another unit, the options and student-evaluator mapping do not persist because the users (both evaluators and students) might be different.
2. Who can set up these advanced options for the assignment? Anyone with “Add/Edit Assignment Submission Folders” permission can set up these new options.
3. What happens when a new student enrolls in the unit? Based on the allocation option set for the assignment,
   we map the new student to an evaluator.
4. Group Assignments do not have these additional options.
5. Turnitin can be used to view Similarity reports- but not for marking.

A video of the process is below: