Skip to content

Testing, debugging and quality control

This page deals with testing questions and quality control. This is largely done through the question test functionality.

Question authoring checklist

High-quality question production needs care at each stage.

Minimal requirements

  1. The question name should be meaningful and consistent, i.e. match up to course, section and topic. E.g. 2018ILA-Wk3-Q2: equation of plane.
  2. Is the phrasing of the question clear to students?
  3. Will students know how to input an answer?
  4. Could a "syntax hint" or message in the question help?
  5. Can "validation" help, e.g. by telling students how many significant figures are expected? (See the "numbers" input type.)
  6. Use question variable stubs throughout, to enable efficient random generation. (E.g. define the correct answer in question variables, rather than hard-wiring a specific expression).
  7. Always make sure the question marks the correct answer as correct!
  8. Add question tests for one correct and at least one incorrect variant. (See below.)
  9. Check all options in the question, inputs and PRTs.

Phase 1

  1. Minimal random variants.
  2. Worked solution ("General feedback") reflecting the random variables.
  3. Consider likely mistakes, and add feedback to test for this.
  4. Add at least one question test to test for each eventuality identified above.

Phase 2

Use data obtained from one cycle of attempts by students.

  1. Did the question operate correctly? E.g. were correct answers correctly marked, and incorrect answers rejected?
  2. What did students get wrong? Is there a reason for these answers such as a common misconception? If so, add nodes to the PRTs to test for this and improve feedback.
  3. Add further question tests to test each misconception.
  4. Is there any significant difference between random variants?

Testing for quality control

It is important to test questions to ensure they work correctly. One approach would be for the teacher to repeatedly test them with various random numbers and inputs. But this is inefficient and unreliable.

Question Tests provide an automated mechanism through which the author may establish with confidence that the Potential Response Trees are processing the student answer as expected. They are based on the concept of "unit testing" from software development. Question variables can be included in the tests; indeed these are needed to define test inputs in the context of random values.

Each test assigns values to

  1. any/all of the inputs. These values may, naturally, depend on the question variables.
  2. values for the score and any penalty.
  3. Answer notes from each of the potential response trees.

The teacher can opt to run the question tests from the preview window. STACK automatically takes each test, assigns the values to inputs and attempts to submit it. The results of the last answer note from each potential response tree is compared to that specified by the teacher. Notice that this is a limitation of the system. Specifying the complete route through the potential response tree would be too difficult and would discourage teachers from writing tests.

If the score, penalty and answer note generated by each potential response tree matches that specified by the teacher, then the test will PASS, otherwise the test will FAIL. Note that failure of a test is a mismatch between outputs and expected outputs, not necessarily a sign that something is wrong with the question itself!

The teacher can also examine by hand the outcomes generated by each step of each test, including full feedback generated.

In this way, the teacher can record, within the question itself, how they expect the marking scheme to work for a variety of student answers.

Writing tests

  1. Author and save your question.
  2. From the Question bank, choose the Preview option.
  3. The Preview question window will open. If you have authority to edit the question, then the top right of the question window will contain a link to question tests and deployed variants.... Follow this link.
  4. This page manages both question tests and deployed variants. Initially you will have no tests or deployed variants. At the bottom of this page choose Add a test case...
  5. Specify values for each input. This may use the question variables. The values of these variables will be used for any random variants.
  6. Specify the expected outcomes for each potential response tree. This includes the score, penalty and answer note. Note: currently only the last Answer Note, not the whole path through the potential response tree, is examined. This is a limitation.
  7. Once you have added the test case, STACK will automatically validate and submit these responses and display the outcomes.
  8. You may add as many test cases are you need. It is sensible to add in
    1. The correct response
    2. One example of each distinction you wish to make, i.e. if you have added specific feedback then provide an answer you expect to trigger this.
    3. Some "invalid" responses, especially if these are syntactically-valid expressions. E.g. If the answer is an equation such as , then might be invalid if you have chosen the input option "check types". Adding a test case is useful to confirm this potential problem is caught by the question. Leave the fields empty and the answer note NULL to indicate this.
    4. Add a totally incorrect answer.

Just above the "Question tests" section of this page is a "Send to CAS" button. Pressing this sends the question variables and general feedback to a special page which enables more efficient authoring of the feedback in the context of the values of the variables. You still need to copy this by hand into the question edit form when you are satisfied.

A Moodle administrator can run all of the questions tests within a particular course, or across the whole site by following the links on the STACK admin page.

Maxima evaluation

Note, that while we use simp:false Maxima must "evaluate" the expression prior to the result being used by an input as a test case.

If the raw testcase expression cannot be sent to the CAS, e.g. a missing bracket, then this invalidity will be tested.

However, some evaluations in Maxima do actually more than just replace existing variables with the values from the question variables.

For example, in Maxima try


Notice here, that while simp:false the limit is still evaluated. This is not "simplification". For the full story, please refer to the Maxima docs on the ev command.

In this case, you can prevent evaluation of limits by using an apostrophie in the test case.


STACK-Maxima sandbox

It is very useful to be able to use the desktop Maxima application to test questions. To do this it is very helpful to load all the STACK libraries. Details on how to do this are in the STACK-Maxima sandbox page.


You can set global simplification flags in two places within questions:

  1. Globally in the question.
  2. In each potential response tree.

Regardless of what settings you use here the expressions you enter for inputs in question tests are not simplified. This is necessary. For example, if your question is "what is ?" where {@a@} and {@b@} are randomly generated. You will need to set the question level option simplify:false to prevent the student typing in the sum itself as an answer. Then you will probably need separate tests for the expressions a+b and ev(a+b,simp) to make sure the student hasn't typed in the sum instead of the value of the sum. For this reason, to enable "unsimplified" expressions to be included as question tests we do not simplify test inputs regardless of the options used in the question.

If you have set simplify:true everywhere in your question, and you are only establishing algebraic equivalence of your answers anyway, "un-simplified" expressions as inputs to the tests will not matter.

Next steps

When you are done testing a question which uses randomization, you need to deploy variants of the question.