Skip to content

Unit tests

Unit testing for STACK comes in the following three parts.

  • PHP Unit tests,
  • Maxima unit tests,
  • Test scripts exposed to the question author.

These three mechanisms aim to provide comprehensive testing of STACK. The last category are a compromise, and are designed to expose the results of unit tests to question authors in a reasonably attractive manner to inform them of what each answer test is actually supposed to do. Links to these tests are in the healthcheck page.

STACK uses the moodle-ci continuous integration mechanism via github actions so that all unit tests are triggered when a commit is pushed to github.

PHP Unit tests

Moodle uses PHPUnit for its unit tests. Setting this up and getting it working is a bit of a pain, but you only have to follow the instructions in the Moodle PHPUnit documentation once to get it working.

STACK-specific set-up steps

Once you have executed

php admin/tool/phpunit/cli/init.php

you need to edit the config.php file to add the following configuration information near the end, but before the require_once(dirname(__FILE__) . '/lib/setup.php');. Other options for the platform are linux and linux-optimised.

/* Options for the platform are `linux` and `linux-optimised`. */
define('QTYPE_STACK_TEST_CONFIG_PLATFORM',        'linux');
/* It is essential that the MAXIMAVERSION and MAXIMACOMMAND match.
   That is, you must check that the command executed here really loads
   the version specified in MAXIMAVERSION.  Some unit tests are version
   dependent.  Do not use default.  */
define('QTYPE_STACK_TEST_CONFIG_MAXIMAVERSION',   '5.42.0');
define('QTYPE_STACK_TEST_CONFIG_MAXIMACOMMAND',   'maxima --use-version=5.42.0');
define('QTYPE_STACK_TEST_CONFIG_MAXIMACOMMANDOPT',   '');
define('QTYPE_STACK_TEST_CONFIG_MAXIMACOMMANDSERVER',   'http://pool.home:8080/MaximaPool/MaximaPool');
define('QTYPE_STACK_TEST_CONFIG_CASTIMEOUT',      '20');
define('QTYPE_STACK_TEST_CONFIG_MAXIMALIBRARIES', 'stats, distrib, descriptive, simplex');
define('QTYPE_STACK_TEST_CONFIG_CASDEBUGGING',    '0');
define('QTYPE_STACK_TEST_CONFIG_PLOTCOMMAND',     '');

define('QTYPE_STACK_TEST_CONFIG_CASRESULTSCACHE', 'db');
define('QTYPE_STACK_TEST_CONFIG_CASPREPARSE', 'true');

You should probably copy the settings from Admin -> Plugins -> Question types -> STACK. However, you can use the flexibility to have different configurations of STACK for testing in order to test a new release of Maxima, for example.

If you want to run just the unit tests for STACK, you can use the command

vendor/bin/phpunit --group qtype_stack

To make sure this keeps working, please annotate all test classes with

/**
 * @group qtype_stack
 * @covers class_name
 */

To generate coverage reports you need to install xdebug. Then modify php.ini configuration file to include xdebug.mode=coverage.

Commands are

vendor/bin/phpunit --testsuite qtype_stack_testsuite --coverage-html folder-name 
vendor/bin/phpunit question/type/stack/tests/test.php --coverage-html folder-name

(where folder-name is the folder that you want to contains the report)

If, for some reason, you have the STACK code in your codebase, and you want to run other unit tests on a server without Maxima installed, then you will get an error when you try to install the PHPunit site. You can avoid that by putting

define('QTYPE_STACK_TEST_CONFIG_PLATFORM',        'none');

in your config.php file. This will prevent the install from trying to create maxima-optimised. It will also cause most of the STACK unit tests to be skipped.

Stop resetting the dataroot directory.

In [...]/moodle/lib/phpunit/classes/util.php 

public static function reset_all_data() {

Comment out the line (currently 253).

self::reset_dataroot();

This stops the unit tests from deleting the Maxima image files at each step.

Making the tests faster

The tests will be very slow, because the Moodle PHPUnit integration keeps resetting the database state between each test, so you get no benefit from the cache. To get around that problem, you can use the option to connect to a different database server for the cache. Modify the following to suit your system and put this near the end of your config.php file:

Note you need to make sure the QTYPE_STACK_TEST_CONFIG_CASRESULTSCACHE variable is only defined once.

define('QTYPE_STACK_TEST_CONFIG_CASRESULTSCACHE',   'otherdb');
define('QTYPE_STACK_TEST_CONFIG_CASCACHEDBTYPE',    $CFG->dbtype);
define('QTYPE_STACK_TEST_CONFIG_CASCACHEDBLIBRARY', $CFG->dblibrary);
define('QTYPE_STACK_TEST_CONFIG_CASCACHEDBHOST',    $CFG->dbhost);
define('QTYPE_STACK_TEST_CONFIG_CASCACHEDBNAME',    $CFG->dbname);
define('QTYPE_STACK_TEST_CONFIG_CASCACHEDBUSER',    $CFG->dbuser);
define('QTYPE_STACK_TEST_CONFIG_CASCACHEDBPASS',    $CFG->dbpass);
define('QTYPE_STACK_TEST_CONFIG_CASCACHEDBPREFIX',  $CFG->prefix);

To make sure the CAS cache is cleared after each unit test, revert back to the db settings for QTYPE_STACK_TEST_CONFIG_CASRESULTSCACHE as described above. This will be slow...

Other configuration issues

Moodle overrides the PHP debug message settings. To see errors and warnings, go to

Site administration -> Development -> Debugging

and set the Debug messages option.

Maxima unit tests

Maxima has a unit testing framework called "rtest". One complication is that we need to run tests with and without simplification. To help with this, a batch file is provided to run the unit tests.

\moodle\question\type\stack\stack\maxima\unittests_load.mac

To run this set up the STACK-maxima-sandbox and load STACK's libraries. Then type

load("unittests_load.mac");

The output from these tests is written to .ERR files in \moodle\question\type\stack\stack\maxima\.

Please note that currently, with simplification false, there are a number of false negative results. That is tests appear to fail, but do not. This is because rtest is not designed to run with simp:false, and so does not correctly decide whether things are really the "same" or "different".

Timing the code.

Maxima has a range of functions for code profiling. Put the following at the start of the file.

timer(all)$

This adds all user-defined functions to the timer list.

To time a single command

ev(timer_info(abs_replace), simp);

To profile all user-defined commands execute.

simp:true$
T:timer_info()$

Find those commands actually called (based on T being the matrix above).

S:sublist(rest(args(T)),lambda([a], not(is(third(a)=0))));

Sort by functions called most often.

S:sort(S, lambda([a,b],third(a)>third(b)));

Sort by the time/call

float_time(a):= if a=0 then 0 else first(args(a))$
S:sort(S, lambda([a,b],float_time(second(a))>float_time(second(b))));

Testing ajax specific problems.

You need to output values to the file system, as the display can't manage this. For example,

file_put_contents("/tmp/log.txt", print_r($result, true));

Testing the updated parser in STACK 4.3

In the STACK directory

php cli/casstringtester.php --string="0..1"