lnet_tutor/docs/multiple-choices-and-idk.md

Multiple Choices and "I Don't Know" Option

New Question Type: multiple_choices

Purpose

Allows questions with multiple correct answers (e.g., "Select all that apply").

Format

{
  "id": "q1",
  "type": "multiple_choices",
  "prompt": "Which of the following are mutable data types in Python? (Select all that apply)",
  "choices": [
    { "key": "A", "text": "list" },
    { "key": "B", "text": "tuple" },
    { "key": "C", "text": "dict" },
    { "key": "D", "text": "str" },
    { "key": "E", "text": "set" }
  ],
  "answer": ["A", "C", "E"],
  "partialCredit": true,
  "allowIDK": true,
  "points": 10
}

Scoring

Partial Credit (default):

Score = (Correct Selections / Total Correct Answers) × Points

Example:

• Correct answers: A, C, E (3 total)
• Student selects: A, C (2 correct)
• Score = (2/3) × 10 = 6.67 points

All or Nothing:

"partialCredit": false

• Gets full points only if ALL correct answers selected
• Any mistake = 0 points

With Wrong Selections:

• Penalty for selecting incorrect options
• Score = max(0, (Correct - Wrong) / Total Correct × Points)

Example:

• Student selects: A, C, D (2 correct, 1 wrong)
• Score = (2 - 1) / 3 × 10 = 3.33 points

"I Don't Know" Option

Purpose

• Encourages honest assessment
• Prevents random guessing
• Better measures actual knowledge
• Can be scored differently (0 points vs penalty)

Availability

Automatically added to:

• single_choice questions
• multiple_choices questions
• true_false questions

Not added to:

• essay questions (can leave blank)
• code_simple / code_exercise (can leave blank)

Format

Enable (default):

{
  "type": "single_choice",
  "allowIDK": true,
  "prompt": "What is...?",
  ...
}

Disable:

{
  "type": "single_choice",
  "allowIDK": false,
  "prompt": "What is...?",
  ...
}

UI Behavior

Single Choice:

○ A. Option A
○ B. Option B
○ C. Option C
○ ? I don't know

Multiple Choices:

☐ A. Option A
☐ B. Option B
☐ C. Option C
☐ ? I don't know (if checked, clears other selections)

True/False:

○ True
○ False
○ I don't know

Scoring Rules

"I don't know" selected:

• Treated as incorrect (0 points)
• NOT penalized (better than guessing wrong)
• Honest indicator of knowledge gaps

Comparison:

• Wrong guess: 0 points + false confidence
• "I don't know": 0 points + honest gap identification

Complete Examples

Example 1: Single Choice with IDK

{
  "id": "q1",
  "type": "single_choice",
  "prompt": "Which decorator is used for static methods?",
  "choices": [
    { "key": "A", "text": "@staticmethod" },
    { "key": "B", "text": "@classmethod" },
    { "key": "C", "text": "@property" }
  ],
  "answer": "A",
  "allowIDK": true,
  "points": 5
}

Possible responses:

• "A" → 5 points (correct)
• "B" or "C" → 0 points (incorrect)
• "IDK" → 0 points (honest)

Example 2: Multiple Choices with Partial Credit

{
  "id": "q2",
  "type": "multiple_choices",
  "prompt": "Which are valid Python keywords? (Select all that apply)",
  "choices": [
    { "key": "A", "text": "class" },
    { "key": "B", "text": "function" },
    { "key": "C", "text": "import" },
    { "key": "D", "text": "include" },
    { "key": "E", "text": "def" }
  ],
  "answer": ["A", "C", "E"],
  "partialCredit": true,
  "allowIDK": true,
  "points": 9
}

Possible responses:

• ["A", "C", "E"] → 9 points (all correct)
• ["A", "C"] → 6 points (2/3 correct with partial credit)
• ["A", "C", "D"] → 3 points (2 correct - 1 wrong)
• ["IDK"] → 0 points (honest)

Example 3: True/False with IDK

{
  "id": "q3",
  "type": "true_false",
  "prompt": "Python supports tail call optimization.",
  "answer": false,
  "allowIDK": true,
  "points": 3
}

Possible responses:

• false → 3 points (correct)
• true → 0 points (incorrect)
• "IDK" → 0 points (honest)

Example 4: Disable IDK (Force Answer)

{
  "id": "q4",
  "type": "single_choice",
  "prompt": "What is 2 + 2?",
  "choices": [
    { "key": "A", "text": "3" },
    { "key": "B", "text": "4" },
    { "key": "C", "text": "5" }
  ],
  "answer": "B",
  "allowIDK": false,
  "points": 2
}

UI shows only A, B, C (no IDK option)

Answer Format

Response for single_choice

{
  "questionId": "q1",
  "response": "A"
}
// or
{
  "questionId": "q1",
  "response": "IDK"
}

Response for multiple_choices

{
  "questionId": "q2",
  "response": ["A", "C", "E"]
}
// or
{
  "questionId": "q2",
  "response": ["IDK"]
}

Response for true_false

{
  "questionId": "q3",
  "response": true
}
// or
{
  "questionId": "q3",
  "response": "IDK"
}

Benefits

For Learners:

• Honest self-assessment
• Identify knowledge gaps
• Avoid false confidence from lucky guesses
• Better learning outcome tracking

For Instructors:

• Clearer picture of student knowledge
• Identify commonly unknown topics
• Better curriculum adjustment
• More accurate assessment

Implementation Notes

• IDK option rendered in UI automatically when allowIDK is true
• Selecting IDK clears any other selections (for multiple_choices)
• Scoring treats IDK as incorrect (0 points)
• Analytics can track IDK rate per question
• High IDK rate = topic needs more coverage