STACKドキュメント

ドキュメントホーム | カテゴリインデックス | サイトマップ | 英語のサイトマップ

Testing, debugging and quality control

This page deals with testing questions and quality control. This is largely done through the question test functionality.

We have separate advice on fixing broken questions in a live quiz.

Question authoring checklist

High-quality question production needs care at each stage.

Minimal requirements

  1. The question name should be meaningful and consistent, i.e. match up to course, section and topic. E.g. 2018ILA-Wk3-Q2: equation of plane.
  2. Is the phrasing of the question clear to students?
  3. Will students know how to input an answer?
    • Could a "syntax hint" or message in the question help?
    • Can "validation" help, e.g. by telling students how many significant figures are expected? (See the "numbers" input type.)
  4. Use question variable stubs throughout, to enable efficient random generation. (E.g. define the correct answer in question variables, rather than hard-wiring a specific expression).
  5. Add a meaingful question note which will make sense later, not just a list of randomly generated numbers. This could be an abreviated form of the question together with the answer.
  6. Add question tests for one correct and at least one incorrect variant. (See below.) Always make sure the question marks the correct answer as correct!
  7. Check all options in the question, inputs and PRTs.

Phase 1

  1. Minimal random variants.
  2. Worked solution ("General feedback") reflecting the random variables.
  3. Consider likely mistakes, and add feedback to test for this.
  4. Add at least one question test to test for each eventuality identified above.

Phase 2

Use data obtained from one cycle of attempts by students.

  1. Did the question operate correctly? E.g. were correct answers correctly marked, and incorrect answers rejected?
  2. What did students get wrong? Is there a reason for these answers such as a common misconception? If so, add nodes to the PRTs to test for this and improve feedback.
  3. Add further question tests to test each misconception.
  4. Is there any significant difference between random variants?

Testing for quality control

It is important to test questions to ensure they work correctly. One approach would be for the teacher to repeatedly test them with various random numbers and inputs. But this is inefficient and unreliable.

Question Tests provide an automated mechanism through which the author may establish with confidence that the Potential Response Trees are processing the student answer as expected. They are based on the concept of "unit testing" from software development. Question variables can be included in the tests; indeed these are needed to define test inputs in the context of random values.

Each test assigns values to

  1. any/all of the inputs. These values may, naturally, depend on the question variables.
  2. values for the score and any penalty.
  3. Answer notes from each of the potential response trees.

The teacher can opt to run the question tests from the preview window. STACK automatically takes each test, assigns the values to inputs and attempts to submit it. The results of the last answer note from each potential response tree is compared to that specified by the teacher. Notice that this is a limitation of the system. Specifying the complete route through the potential response tree would be too difficult and would discourage teachers from writing tests.

If the score, penalty and answer note generated by each potential response tree matches that specified by the teacher, then the test will PASS, otherwise the test will FAIL. Note that failure of a test is a mismatch between outputs and expected outputs, not necessarily a sign that something is wrong with the question itself!

The teacher can also examine by hand the outcomes generated by each step of each test, including full feedback generated.

In this way, the teacher can record, within the question itself, how they expect the marking scheme to work for a variety of student answers.

Writing tests

  1. Author and save your question.
  2. From the Question bank, choose the Preview option.
  3. The Preview question window will open. If you have authority to edit the question, then the top right of the question window will contain a link to question tests and deployed variants.... Follow this link.
  4. This page manages both question tests and deployed variants. Initially you will have no tests or deployed variants. Choose Add a test case...
  5. Specify values for each input. This may use the question variables. The values of these variables will be used for any random variants.
  6. Specify the expected outcomes for each potential response tree. This includes the score, penalty and answer note. Note: currently only the last Answer Note, not the whole path through the potential response tree, is examined. This is a limitation.
  7. Once you have added the test case, STACK will automatically validate and submit these responses and display the outcomes.
  8. You may add as many test cases are you need. It is sensible to add in the following.
    1. The correct response. There is a button which will copy the expression used as the "Teacher's answer" in the input as a basis for a test case to help create this test.
    2. One example of each distinction you wish to make, i.e. if you have added specific feedback then provide an answer you expect to trigger this.
    3. Some "invalid" responses, especially if these are syntactically-valid expressions. E.g. If the answer is an equation such as \(y=2x+1\), then \(2x+1\) might be invalid if you have chosen the input option "check types". Adding a test case is useful to confirm this potential problem is caught by the question. Leave the fields empty and the answer note NULL to indicate this.
    4. Add a totally incorrect answer.
  9. If you leave the penalty field blank it will assume you mean the default penalty for the question.

On the question testing page is a "Send to CAS" button. Pressing this sends the question variables and general feedback to a special page which enables more efficient authoring of the feedback in the context of the values of the variables. You still need to copy this by hand into the question edit form when you are satisfied.

A Moodle administrator can run all of the questions tests within a particular course, or across the whole site by following the links on the STACK admin page. It is useful do to this after upgrading the STACK code on the server to identify any test cases which have changed.

Test case construction and Maxima evaluation

Test cases are always written assuming simp:false regardless of the option set elsewhere. If you want to construct a simplified test case then wrap this in ev(... , simp) to simplify the expression generating the test case. This behaviour is required to enable construction of unsimplified test cases.

You can (and should) constuct test cases based on invalid expressions. If the raw testcase expression cannot be sent to the CAS, e.g. a missing bracket, then this invalidity will be tested.

While test case construction uses simp:false Maxima must "evaluate" the expression prior to the result being used by an input as a test case. This will replace variables by their values. E.g. the typical case is to define a variable such as ta as the teacher's answer in the question variables field and use this throughout the question. This answer will either be simplified, or not, when the question variables are evaluated. To construct a test case using the teacher's answer use ta as the test case input.

It is easier to create a number of variables in the question variables field for wrong answers, e.g. wa1, wa2, and construct the test cases in the question variables in advance.

For matrix inputs the test case must be a correct Maxima statement, e.g. matrix([1,2],[2,3]). There is currently no option to construct test cases by filling in individual input boxes here, which restricts the ability to create test cases of individual syntactically invalid matrix entries. Similarly, for the textarea and equivalence reasoning input types the test case should be constructed as a list, just as the teacher's answer field is constructed.

Some evaluations in Maxima do actually more than just replace existing variables with the values from the question variables. For example, in Maxima try

simp:false;
f:x*sin(1/x);
limit(f,x,0);

Notice here, that while simp:false the limit is still evaluated. This is not "simplification". For the full story, please refer to the Maxima docs on the ev command.

In this case, you can prevent evaluation of limits by using an apostrophie in the test case construction to prevent evaluation.

simp:false;
f:x*sin(1/x);
'limit(f,x,0);

Test case construction and MCQ

Remember that MCQ input types return a CAS expression. However, you must ensure each testcase actually corresponds to an option provided to the student, otherwise STACK will generate a run-time error. Hence you cannot construct a test case from the list provided as the "Teacher's answer" in these input types!

It is sensible to use the helper functions. E.g. for a radio/dropdown use

tc1:first(mcq_correct(ta));

For the checkbox type you will need the whole list.

tc1:mcq_correct(ta);

Test case construction and numerical precision.

You can construct test cases using the functions such as dispdp to create a test-case input with trailing zeros. This is neeeded if the input, or answer test, is testing for a minimum number of decimal places or significant figures.

STACK-Maxima sandbox

It is very useful to be able to use the desktop Maxima application to test questions. To do this it is very helpful to load all the STACK libraries. Details on how to do this are in the STACK-Maxima sandbox page.

Simplification

You can set global simplification flags in two places within questions:

  1. Globally in the question.
  2. In each potential response tree.

Regardless of what settings you use here the expressions you enter for inputs in question tests are not simplified. This is necessary. For example, if your question is "what is \({@a@}+{@b@}\)?" where {@a@} and {@b@} are randomly generated. You will need to set the question level option simplify:false to prevent the student typing in the sum itself as an answer. Then you will probably need separate tests for the expressions a+b and ev(a+b,simp) to make sure the student hasn't typed in the sum instead of the value of the sum. For this reason, to enable "unsimplified" expressions to be included as question tests we do not simplify test inputs regardless of the options used in the question.

If you have set simplify:true everywhere in your question, and you are only establishing algebraic equivalence of your answers anyway, "un-simplified" expressions as inputs to the tests will not matter.

Next steps

When you are done testing a question which uses randomization, you need to deploy variants of the question.

ドキュメントホーム | カテゴリインデックス | サイトマップ | 英語のサイトマップ