Skip to content

Testing, debugging and quality control

This page deals with testing questions and quality control. An authoring workflow is described separately.

We also have separate advice on fixing broken questions in a live quiz.

Testing for quality control

It is important to test questions to ensure they work correctly. One approach would be for the teacher to repeatedly test them with various random numbers and inputs. But this is inefficient and unreliable.

Question Tests provide an automated mechanism through which the author may establish with confidence that the Potential Response Trees are processing the student answer as expected, and we get the same behaviour for each random variant. Question Tests are based on the concept of "unit testing" from software development. Question variables can be included in the tests; indeed these are needed to define test inputs in the context of random values.

Each Question Test assigns values to

  1. any/all of the inputs. These values may, naturally, depend on the question variables.
  2. values for the score and any penalty.
  3. Answer notes from each of the potential response trees.

STACK automatically takes each test, assigns the values to inputs and attempts to submit it. Note that failure of a test is a mismatch between outputs and expected outputs, not always a sign that something is wrong with the question itself!

We strongly recommend that every question has a test case based on the teacher's answers.

The teacher can also examine by hand the outcomes generated by each step of each test, including full feedback generated. In this way, the teacher can record, within the question itself, how they expect the marking scheme to work for a variety of student answers. This is also useful when reading other people's questions.

Writing tests

  1. Author and save your question.
  2. Navigate to the "STACK question dashboard". This page manages both question tests and deployed variants. Initially you will have no tests or deployed variants.
  3. We strongly recommend that every question has a test case based on the teacher's answers. Typically (except, perhaps with survey items) this will get a score of 1 from each PRT. To add such a test case choose Add test case assuming the teacher's answer gets full marks, and review the actual output carefully.
  4. Choose Add a test case. to add more tests. We strongly recommend that every question has a test case based on one incorrect response (proving the PRT does not award marks to all inputs!)
  5. Specify values for each input. This may use the question variables. The values of these variables will be used for any random variants, but regardless of the question options simplification is not used in test case construction (see below).
  6. Specify the expected outcomes for each potential response tree. This includes the score, penalty and answer note.
  7. Scores and penalties must be floats (not, e.g. 2/3) and are rounded to three decimal places for testing purposes.
  8. See below for answer note construction.
  9. Once you have added the test case, STACK will automatically validate and submit these responses and display the outcomes.
  10. You may add as many test cases are you need. It is sensible to add in the following.
  11. The teacher's response (which is typically correct). There is a button which will copy the expression used as the "Teacher's answer" in the input as a basis for a test case to help create this test.
  12. Add one totally incorrect answer.
  13. One example of each distinction you wish to make, i.e. if you have added specific feedback then provide an answer you expect to trigger this.
  14. Some "invalid" responses, especially if these are syntactically-valid expressions. E.g. If the answer is an equation such as , then might be invalid if you have chosen the input option "check types". Adding a test case is useful to confirm this potential problem is caught by the question. Leave the fields empty and the expected answer note NULL to indicate this.
  15. If you leave the penalty field blank it will assume you mean the default penalty for the question.
  16. If you set either the score or penalty field to -1 then the test will pass on any value of that field, effectively ignoring the field for the purposes of this test case. This is not recommended (as you're not testing anymore!).

Normally test cases are constructed taking into account values of the question variables, so test-case construction can reflect any random variants. If you start the expected value of an input with the tag RAW: (case sensitive) then the remainder of your input will be used as a raw string. E.g. if your test case is RAW:2 x then your input test case will be 2 x. Note, this feature does not evaluate the expression further, and values of question variables will not be used as part of constructing the input for this test case. However, the potential response tree will have access to the question variables later. For example, if n:42 in the question variables then typically student's who input n will get a validation error. RAW:n allows you to create a test case, which will be displayed as n but later the PRTs will pick up the question variables. This feature is intended to test invalid input rather than evaluation of test cases constructed from the random variables. We can't have it both ways!

If you start your test case with the tag CT: (case sensitive) then the remainder of your input will be evaluated as a castext string. E.g. if your test case is CT:{#a#}{#v#} then the castext {#a#}{#v#} will be evaluated, and the values of variables a and v placed next to each other to create an input string. This feature can be used to test input settings, such as insert stars, is working leading to a "score" state rather than an invalid state. Please do not use CT:{@a@} which will generate LaTeX output! For test-case construction we need to create Maxima syntax, not LaTeX.

On the STACK question dashboard is a link "Send general feedback to the CAS". This sends the question variables and general feedback to a special page which enables more efficient authoring of the feedback in the context of the values of the variables. This page is also useful for test-case construction.

A Moodle administrator can bulk-test all of the questions tests within a particular course, or across the whole site by following the links on the STACK admin page. It is useful to do this after upgrading the STACK code on the server to identify any test cases which have changed.

Test cases can include a meaningful description of up to 255 characters. This field is a simple string, and is not castext.

Answer note expectation construction

STACK allows you to confirm the route through the potential response tree by specifying expected answer notes.

When constructing test cases the teacher separates expected notes with | characters.

The test passes if and only if the PRT produces all the expected notes in the specified order. We condone any extra notes from the PRT which don't appear in the test case expectation.

For example, consider the expectation prt1-1-F | prt1-3-F | prt1-4-F from PRT prt1. Note in this example, there is no expectation from node 2. Hence, all the following answer notes from the PRT will pass the test.

prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-F
prt1-1-F | prt1-2-F | prt1-3-F | prt1-4-F
prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-F | prt1-5-F | prt1-6-F
prt1-0-F | prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-F

Notes:

  • Empty answer notes are not permitted, but the special note () indicates any answer note is accepted (rather than using the any string).
  • The special note NULL is used for an expectation that the PRT did not fire (e.g. invalid input).
  • The special note ???-bail (e.g. prt1-bail) is used to indicate the tree attempted to execute but then stopped and "bailed out" during evaluation of the feedback variables, and you can test for this. (Do not anchor this note.)
  • Some answer tests generate notes which depend on the specific input. For this reason, when automatically creating test case expectations from particular inputs the notes from the answer test are not included. You can choose to include these notes in the test case expectation manually.
  • We anchor these notes to specify an expected start/end note. Use the range notations: [...] to expect both ends fixed; (...] specifies only the final note; [...) specifies the start note; (...) does not expect a particular start/end note.
  • (...]is the default if no anchor wrapping is defined (this retains back compatibility).
  • Expected test cases are limited to 1000 characters. (Since tags are typically each like prt1-1-F | (10 chars), that's a lot of checkable nodes.)

Hence, with the anchored expectation [ prt1-1-F | prt1-3-F | prt1-4-F ] the following will pass as tests.

prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-F
prt1-1-F | prt1-2-F | prt1-3-F | prt1-4-F

However this result will fail, as prt1-4-F is expected to be the last note.

prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-F | prt1-5-F | prt1-6-F

Legacy test case construction (before v4.13.0)

Before STACK 4.13.0, only the last answer note from each potential response tree was compared to that specified by the teacher. Older questions with legacy question tests will continue to be tested in this way, and legacy test cases will behave as before. (New questions, with new-style tests, will give rise to failed question tests. This is a failure of the test, not necessarily of the question: you will need to review, and update the question tests if using newer questions on older STACK installations, sorry.)

Test case construction and Maxima evaluation

Test cases are always written assuming simp:false regardless of the option set elsewhere. If you want to construct a simplified test case then wrap this in ev(... , simp) to simplify the expression generating the test case. This behaviour is required to enable construction of un-simplified test cases.

Test cases are always written using the period . as the decimal separator. This corresponds to strict Maxima syntax, which teachers should always use.

You can (and should) construct test cases based on invalid expressions. If the raw test-case expression cannot be sent to the CAS, e.g. a missing bracket, then this invalidity will be tested.

While test case construction uses simp:false Maxima must "evaluate" the expression prior to the result being used by an input as a test case. This will replace variables by their values. E.g. the typical case is to define a variable such as ta as the teacher's answer in the question variables field and use this throughout the question. This answer will either be simplified, or not, when the question variables are evaluated. To construct a test case using the teacher's answer use ta as the test case input.

It is easier to create a number of variables in the question variables field for wrong answers, e.g. wa1, wa2, and construct the test cases in the question variables in advance.

For matrix inputs the test case must be a correct Maxima statement, e.g. matrix([1,2],[2,3]). There is currently no option to construct test cases by filling in individual input boxes here, which restricts the ability to create test cases of individual syntactically invalid matrix entries. Similarly, for the textarea and equivalence reasoning input types the test case should be constructed as a list, just as the teacher's answer field is constructed.

Some evaluations in Maxima do actually more than just replace existing variables with the values from the question variables. For example, in Maxima try

simp:false;
f:x*sin(1/x);
limit(f,x,0);

Notice here, that while simp:false the limit is still evaluated. This is not "simplification". For the full story, please refer to the Maxima docs on the ev command.

In this case, you can prevent evaluation of limits by using an apostrophe in the test case construction to prevent evaluation.

simp:false;
f:x*sin(1/x);
'limit(f,x,0);

Test cases for specific question types

Test case construction and MCQ

Remember that MCQ input types return a CAS expression. However, you must ensure each test-case actually corresponds to an option provided to the student, otherwise STACK will generate a runtime error. Hence you cannot construct a test case from the list provided as the "Teacher's answer" in these input types!

It is sensible to use the helper functions. E.g. for a radio/dropdown use

tc1:first(mcq_correct(ta));

For the checkbox type you will need the whole list.

tc1:mcq_correct(ta);

Test case construction and numerical precision

You can construct test cases using the functions such as dispdp to create a test-case input with trailing zeros. This is needed if the input, or answer test, is testing for a minimum number of decimal places or significant figures.

Test case construction and decimal separators

The decimal separator option (e.g. . or ,) is a very thin layer based on the student input. The teacher must always use a . (full stop) as the decimal separator in question variables. Consistent with this, all test-case construction must use a . (full stop) as the decimal separator. This means it's hard to test the functionality of the decimal separator option (sorry), but otherwise there is genuine confusion in the internal logic about when to assume a , is a decimal separator or a list separator. Also, if you change this option in the question you do not need to change all your test cases.

Testing values of variables

If there are any runtime errors in the Question Variables then question tests will fail (even if the errors do not affect the outcomes of the particular test). Question variables should be runtime error free.

Runtime errors in the Feedback variables of a potential response tree are ignored by the testing mechanism. This field can be used as a guard clause to the PRT. E.g. evaluating a rational expression might result in division by zero which can be ignored.

STACK provides a special function s_assert(ex1, ex2) for testing, which can be used in the question variables and feedback variables. If is(ex1=ex2) does not evaluate to true then this function throws a maxima error message. This can be used to create a runtime error and prevent a question, and a particular variant, being used.

For example, if you have an expression 1/n and the variable n is randomly generated you need to prevent a random version being zero. In this case put the following in the question variables.

s_assert(is(n=0), false);

This test will throw an error in the question variables when n is zero.

In many situations this kind of test creation will be simpler than mapping onto student inputs.

  1. If you place s_assert in the question variables then any run-time errors will stop the question being used by students. These errors will appear when you bulk-test the question variants.
  2. If you place s_assert in the feedback variables of a PRT then any run-time errors will be ignored.

Note, the purpose of s_assert is unit testing of question variables within question testing.

STACK-Maxima sandbox

It is very useful to be able to use the desktop Maxima application to test questions. To do this it is very helpful to load all the STACK libraries. Details on how to do this are in the STACK-Maxima sandbox page.

Simplification

You can set global simplification flags in two places within questions:

  1. Globally in the question.
  2. In each potential response tree.

Regardless of what settings you use here the expressions you enter for inputs in question tests are not simplified. This is necessary. For example, if your question is what is \({@a@}+{@b@}\)? where {@a@} and {@b@} are randomly generated. You will need to set the question level option simplify:false to prevent the student typing in the sum itself as an answer. Then you will probably need separate tests for the expressions a+b and ev(a+b,simp) to make sure the student hasn't typed in the sum instead of the value of the sum. For this reason, to enable "unsimplified" expressions to be included as question tests we do not simplify test inputs regardless of the options used in the question.

(If you have set simplify:true everywhere in your question, and you are only establishing algebraic equivalence of your answers anyway, "unsimplified" expressions as inputs to the tests will not matter.)

Next steps

When you are done testing a question which uses randomization, you need to deploy variants of the question.