Feedback

The purpose of STACK is to assess students' answers to mathematical questions, and on the basis of the properties we establish to assign feedback.

This document describes the ways STACK provides feedback to students. It will not go into much depth on creating the feedback, which is documented under potential response trees.

Types of assessment

• Formative assessment is to support and inform students' learning. Feedback here could be qualitative, e.g. written comments tailored to the student's answer and designed to help them improve their performance on the task.
• Summative assessment is to establish the achievement of the student. In mathematics, summative feedback is most often quantitative, either a mark or a percentage.
• Evaluative assessment is to measure the effectiveness of the teaching or the assessment of students. Such assessments could have quality enhancement or quality audit functions. See reporting. The ability to automatically generate data about an individual student or across a cohort is one particular strength of CAA, potentially enabling regular, detailed evaluative assessment.

In STACK multi-part questions there is a complete separation between two important components.

1. a list of inputs
2. a list of potential response trees

Specific feedback is associated with each input and each potential response tree. Typically, it is placed in the Specific feedback section, as that gives Moodle more control over when it is shown. However, the feedback tags can be positioned anywhere within the question text.

Validation

Before an input is available to a potential response trees, it must be validated. In particular, at each attempt, each input is assigned a status.

1. NULL, which indicates the field has not been previously given a value by the student, or the field is now empty since the student has deleted an answer.
2. new, indicates an answer has not been validated or assessed, but has been changed from a previous attempt.
3. invalid, which indicates that the field is not valid.
4. valid, a response which is valid but not scored.
5. score. In this case, the answer is available to any potential response tree requiring it.

Normally a student will view a displayed form of their expression and submit it again. This default behaviour is inappropriate for multiple choice/selection interactions, and can be changed for each input using the option "Student must verify". Whether the student's answer is echoed back and shown is controlled by a separate option "Show the validation". Validation errors are always shown.

Whether a string entered by the student is valid or invalid does not depend on the question. I.e. there should be a consistent mechanism for what constitutes a syntactically valid expression. However, in practice things are not quite so clean! Some input options do affect validity, such as forbid floats. Some symbols, e.g. $i$ and $j$ change meaning in different contexts, e.g. $\sqrt{-1}$ or vector components. See details about options.

Potential response trees

Each potential response tree returns three outcomes:

1. a numerical score,
2. text for the students,
3. an answer note for use by the teacher during reporting.

These correspond approximately to summative, formative and evaluative functions of assessment respectively. The general feedback (known as worked solution in previous versions) is fixed and may not depend on the student's answers. Hence it is not considered to be feedback to the student's work in the strict sense. However, it remains a very useful outcome for students.

The amount of feedback available in each question is governed by the question behaviours.

Numerical score

Each potential response tree calculates a numerical score between 0 and 1. This is then multiplied by the question value for each potential response tree. The final score for the question is the sum over all potential response trees.

The numerical scores are assembled by traversing each potential response tree.

• Each branch of each node can add, subtract or set an absolute, score.
• The outcome at the end should be between 0 and 1. If the score, $s$, lies outside this range it is taken to be $\min(\max(s,0),1)$ to bring it within range, then it is scaled by multiplying by the question value for that potential response tree.
• A "penalty" may also set in the potential response tree for this attempt. Normally the penalty field in each branch of the potential response tree is empty, in which case the question level penalty value is used. However, these fields are useful to remove any penalty for this outcome, by setting it to zero explicitly.
• After the whole tree has been traversed, if the score is 1 then the penalty is always assigned to 0.

STACK adjusts the score for each potential response tree, based on the number of valid, different attempts. The penalty scheme deducts from the score a small amount (default is $0.1=10\%$) for each different and valid attempt which is not completely correct. It is designed to reward persistence and diligence when students initially get a question wrong, but they try again.

It works in the following way. For each attempt $k$, we let

• $s_k$ be the score from the potential response tree.
• $p_k$ be the "penalty" as follows:
• If $s_k=1$ then $p_k=0$, else
• If the penalty $p$ set in the last branch traversed before exiting the potential response tree is not NULL then $p_k=p$, else
• $p_k$ is the penalty set in the question options, (default $0.1=10\%$ ).

The default penalty scheme takes the maximum score for each attempt, so that by accruing further penalties a student may never be worse off.

To be specific

1. Let $(s_i,p_i)$ for $i=1,\cdots n$ be the list of scores and penalties for a particular potential response tree, for each different valid attempt.
2. The score for attempt $k$ is defined to be

$$\text{Question value} \times \max\left\{ s_i-\sum_{j=1}^i p_j,\ i=1,\cdots k \right\}.$$

Notice that this is purely a function of a list of (score, penalty) pairs.

The score for that attempt is the sum of the marks for each potential response tree once penalties have been deducted from each tree.

Text for the student

The text-based feedback for students is a concatenation of the following elements.

• Answer test feedback. Many of the answer tests generate feedback of their own. This can be suppressed using the quiet option. While this feedback is often not needed, it would be very difficult for the teacher to re-create this.
• Bespoke feedback. Each branch of the potential response trees generates some feedback.
• Generic feedback. Once the potential response trees has been traversed and all feedback assigned, the score is used to generate some generic feedback. If the raw score equals $0$ then the default feedback is Incorrect answer. If the raw score equals $1$ then the default feedback is Correct answer, well done. Otherwise the generic feedback is Your answer is partially correct. These strings can be modified in the options.