STACK Documentation

Documentation home | Category index | Parent | Site map

Testing, debugging and quality control.

Computer aided assessment of mathematics works in the following phases.

  1. Authoring,
  2. Testing and
  3. Deploying questions.
  4. Adding questions to a quiz and use by students.
  5. Reporting and statistical analysis.

Testing for quality control

It is important to test questions to ensure they work correctly. One approach would be for the teacher to repeatedly test them with various random numbers and inputs. But this is inefficient and unreliable.

Question Tests provide an automated mechanism through which the author may establish with confidence that the Potential Response Trees are processing the student answer as expected. They are based on the concept of "unit testing" from software development. Question variables can be included in the tests, indeed these are needed to define test inputs in the context of random values.

Each test assigns values to

  1. any/all of the inputs. These values may, naturally, depend on the question variables.
  2. values for the score and any penalty.
  3. Answer notes from each of the potential response trees.

The teacher can opt to run the question tests from the preview window. STACK automatically takes each test, assigns the values to inputs and attempts to submit it. The results of the last answer note from each potential response tree is compared to that specified by the teacher. Notice this is a limitation of the system. Specifying the complete route through the potential response tree would be too difficult and would discourage teachers from writing tests.

If the score, penalty and answer note generated by each potential response tree matches that specified by the teacher, then the test will PASS, otherwise the test will FAIL. Note that failure of a test is a mismatch between outputs and expected outputs, not necessarily a sign that something is wrong with the question itself!

The teacher can also examine by hand the outcomes generated by each step of each test, including full feedback generated.

In this way, the teacher can record, within the question itself, how they expect the marking scheme to work for a variety of student answers.

Writing tests

  1. Author and save your question.
  2. From the Question bank, choose the Preview option.
  3. The Preview question window will open. If you have authority to edit the question, then the top right of the question window will contain a link to Run the question tests.... Follow this link.
  4. This page manages both question tests and deployed variants. Initially you will have no tests or deployed variants. At the bottom of this page choose Add a test case...
  5. Specify values for each input. This may use the question variables. The values of these variables will be used for any random versions.
  6. Specify the expected outcomes for each potential response tree. This includes the score, penalty and answer note. Note: currently only the last Answer Note, not the whole path through the potential response tree, is examined. This is a limitation.
  7. Once you have added the test case, STACK will automatically validate and submit these responses and display the outcomes.
  8. You may add as many test cases are you need. It is sensible to add in
    1. The correct response
    2. One example of each distinction you wish to make. I.e. if you have added specific feedback then provide an answer you expect to trigger this.
    3. Some "invalid" responses, especially if these are syntactically valid expressions. E.g. If the answer is an equation such as \(y=2x+1\), then \(2x+1\) might be invalid if you have chosen the input option "check types". Adding a test case is useful to confirm this potential problem is caught by the question. Leave the fields empty and the answer note NULL to indicate this.
    4. Add a totally incorrect answer.

Just above the "Question tests" section of this page is a "Send to CAS" button. Pressing this sends the question variables and general feedback to a special page which enables more efficient authoring of the feedback in the context of the values of the variables. You still need to copy this by hand into the question edit form when you are satisfied.

STACK-Maxima sandbox

It is very useful to be able to use the desktop Maxima application to test questions. To do this it is very helpful to load all the STACK libraries. Details on how to do this are in the STACK-Maxima sandbox page.


You can set global simplification flags in two places within questions:

  1. Globally in the question.
  2. In each potential response tree.

Regardless of what settings you use here the expressions you enter for inputs in question tests are not simplified. This is necessary. For example, if your question is "what is \(@a@+@b@\)?" where @a@ and @b@ are randomly generated. You will need to set the question level option simplify:false to prevent the student typing in the sum itself as an answer. Then you will probably need separate tests for the expressions a+b and ev(a+b,simp) to make sure the student hasn't typed in the sum instead of the value of the sum. For this reason, to enable "unsimplified" expressions to be included as question tests we do not simplify test inputs regardless of the options used in the question.

If you have set simplify:true everywhere in your question, and you are only establishing algebraic equivalence of your answers anyway, "unsimplified" expressions as inputs to the tests will not matter.

Next steps

When you are done testing a question which uses randomization, you need to deploy variants of the question.

Documentation home | Category index | Parent | Site map