Skip to content

Testing, debugging and quality control

This page deals with testing questions and quality control. This is largely done through the question test functionality.

High-quality question production needs care at each stage. An authoring workflow is described separately.

We have separate advice on fixing broken questions in a live quiz.

Testing for quality control

It is important to test questions to ensure they work correctly. One approach would be for the teacher to repeatedly test them with various random numbers and inputs. But this is inefficient and unreliable.

Question Tests provide an automated mechanism through which the author may establish with confidence that the Potential Response Trees are processing the student answer as expected. They are based on the concept of "unit testing" from software development. Question variables can be included in the tests; indeed these are needed to define test inputs in the context of random values.

Each test assigns values to

  1. any/all of the inputs. These values may, naturally, depend on the question variables.
  2. values for the score and any penalty.
  3. Answer notes from each of the potential response trees.

The teacher can opt to run the question tests from the preview window. STACK automatically takes each test, assigns the values to inputs and attempts to submit it. The results of the last answer note from each potential response tree is compared to that specified by the teacher. Notice that this is a limitation of the system. Specifying the complete route through the potential response tree would be too difficult and would discourage teachers from writing tests.

If the score, penalty and answer note generated by each potential response tree matches that specified by the teacher, then the test will PASS, otherwise the test will FAIL. Note that failure of a test is a mismatch between outputs and expected outputs, not necessarily a sign that something is wrong with the question itself!

The teacher can also examine by hand the outcomes generated by each step of each test, including full feedback generated.

In this way, the teacher can record, within the question itself, how they expect the marking scheme to work for a variety of student answers.

Writing tests

  1. Author and save your question.
  2. From the Question bank, choose the Preview option.
  3. The Preview question window will open. If you have authority to edit the question, then the top right of the question window will contain a link to question tests and deployed variants.... Follow this link.
  4. This page manages both question tests and deployed variants. Initially you will have no tests or deployed variants. Choose Add a test case...
  5. Specify values for each input. This may use the question variables. The values of these variables will be used for any random variants.
  6. Specify the expected outcomes for each potential response tree. This includes the score, penalty and answer note.
  7. Currently only the last Answer Note, not the whole path through the potential response tree, is examined. This is a limitation.
  8. Scores and penalties are rounded to three decimal places for testing purposes.
  9. Once you have added the test case, STACK will automatically validate and submit these responses and display the outcomes.
  10. You may add as many test cases are you need. It is sensible to add in the following.
    1. The correct response. There is a button which will copy the expression used as the "Teacher's answer" in the input as a basis for a test case to help create this test.
    2. One example of each distinction you wish to make, i.e. if you have added specific feedback then provide an answer you expect to trigger this.
    3. Some "invalid" responses, especially if these are syntactically-valid expressions. E.g. If the answer is an equation such as , then might be invalid if you have chosen the input option "check types". Adding a test case is useful to confirm this potential problem is caught by the question. Leave the fields empty and the answer note NULL to indicate this.
    4. Add a totally incorrect answer.
  11. If you leave the penalty field blank it will assume you mean the default penalty for the question.

If you start your test case with the tag RAW: (case sensitive) then the remainder of your input will be used as a raw string. E.g. if your test case is RAW:2 x then your input test case will be 2 x. Note, this feature does not evaluate the expression further, and values of question variables will not be used.

If you start your test case with the tag CT: (case sensitive) then the remainder of your input will be evaluated as a castext string. E.g. if your test case is CT:{#a#}{#v#} then the castext {#a#}{#v#} will be evaluated, and the values of variables a and v placed next to each other to create an input string. This feature can be used to test input settings, such as insert stars, is working leading to a "score" state rather than an invalid state.

On the question testing page is a "Send to CAS" button. Pressing this sends the question variables and general feedback to a special page which enables more efficient authoring of the feedback in the context of the values of the variables. You still need to copy this by hand into the question edit form when you are satisfied.

A Moodle administrator can run all of the questions tests within a particular course, or across the whole site by following the links on the STACK admin page. It is useful to do this after upgrading the STACK code on the server to identify any test cases which have changed.

Test cases can include a meaningful description of up to 255 characters. This field is a simple string, and is not castext.

Test case construction and Maxima evaluation

Test cases are always written assuming simp:false regardless of the option set elsewhere. If you want to construct a simplified test case then wrap this in ev(... , simp) to simplify the expression generating the test case. This behaviour is required to enable construction of un-simplified test cases.

Test cases are always written using the period . as the decimal separator. This corresponds to strict Maxima syntax, which teachers should always use.

You can (and should) construct test cases based on invalid expressions. If the raw testcase expression cannot be sent to the CAS, e.g. a missing bracket, then this invalidity will be tested.

While test case construction uses simp:false Maxima must "evaluate" the expression prior to the result being used by an input as a test case. This will replace variables by their values. E.g. the typical case is to define a variable such as ta as the teacher's answer in the question variables field and use this throughout the question. This answer will either be simplified, or not, when the question variables are evaluated. To construct a test case using the teacher's answer use ta as the test case input.

It is easier to create a number of variables in the question variables field for wrong answers, e.g. wa1, wa2, and construct the test cases in the question variables in advance.

For matrix inputs the test case must be a correct Maxima statement, e.g. matrix([1,2],[2,3]). There is currently no option to construct test cases by filling in individual input boxes here, which restricts the ability to create test cases of individual syntactically invalid matrix entries. Similarly, for the textarea and equivalence reasoning input types the test case should be constructed as a list, just as the teacher's answer field is constructed.

Some evaluations in Maxima do actually more than just replace existing variables with the values from the question variables. For example, in Maxima try

simp:false;
f:x*sin(1/x);
limit(f,x,0);

Notice here, that while simp:false the limit is still evaluated. This is not "simplification". For the full story, please refer to the Maxima docs on the ev command.

In this case, you can prevent evaluation of limits by using an apostrophie in the test case construction to prevent evaluation.

simp:false;
f:x*sin(1/x);
'limit(f,x,0);

Test case construction and MCQ

Remember that MCQ input types return a CAS expression. However, you must ensure each testcase actually corresponds to an option provided to the student, otherwise STACK will generate a runtime error. Hence you cannot construct a test case from the list provided as the "Teacher's answer" in these input types!

It is sensible to use the helper functions. E.g. for a radio/dropdown use

tc1:first(mcq_correct(ta));

For the checkbox type you will need the whole list.

tc1:mcq_correct(ta);

Test case construction and numerical precision

You can construct test cases using the functions such as dispdp to create a test-case input with trailing zeros. This is neeeded if the input, or answer test, is testing for a minimum number of decimal places or significant figures.

Test case construction and decimal separators

The decimal separator option (e.g. . or ,) is a very thin layer based on the student input. The teacher must always use a . (full stop) as the decimal separator in question variables. Consistent with this, all test-case construction must use a . (full stop) as the decimal separator. This means it's hard to test the functionality of the decimal separator option (sorry), but otherwise there is genuine confusion in the internal logic about when to assume a , is a decimal separator or a list separator. Also, if you change this option in the question you do not need to change all your test cases.

Testing values of variables

STACK provides a special function s_assert(ex1, ex2) which can be used in the question variables and feedback variables. If is(ex1=ex2) does not evaluate to true then this function throws a maxima error message. This can be used to create a run-time error and prevent a question, and a particular variant, being used.

For example, if you have an expression 1/n and the variable n is randomly generated you need to prevent a random version being zero. In this case put the following in the question variables.

s_assert(is(n=0), false);

This test will throw an error when n is zero.

In many situations this kind of test creation will be simpler than mapping onto student inputs.

STACK-Maxima sandbox

It is very useful to be able to use the desktop Maxima application to test questions. To do this it is very helpful to load all the STACK libraries. Details on how to do this are in the STACK-Maxima sandbox page.

Simplification

You can set global simplification flags in two places within questions:

  1. Globally in the question.
  2. In each potential response tree.

Regardless of what settings you use here the expressions you enter for inputs in question tests are not simplified. This is necessary. For example, if your question is "what is ?" where {@a@} and {@b@} are randomly generated. You will need to set the question level option simplify:false to prevent the student typing in the sum itself as an answer. Then you will probably need separate tests for the expressions a+b and ev(a+b,simp) to make sure the student hasn't typed in the sum instead of the value of the sum. For this reason, to enable "unsimplified" expressions to be included as question tests we do not simplify test inputs regardless of the options used in the question.

If you have set simplify:true everywhere in your question, and you are only establishing algebraic equivalence of your answers anyway, "un-simplified" expressions as inputs to the tests will not matter.

Next steps

When you are done testing a question which uses randomization, you need to deploy variants of the question.