Existing reminders
Keep sending the usual reminders.
Evaluation example
Decide what you
need to find out.
Would an additional reminder help an education program fill more of its scheduled seats?
If attendance rises after a change, something else may have caused it. Plan a fair comparison before deciding whether to use the reminder more widely.
The question
Is the additional reminder worth using?
Measure attendance at the sessions included in the study. Record who was assigned the extra reminder, whether it was read and whether the person attended as separate things.
Random
assignment
assignment
One additional reminder
Send one extra reminder and keep the other messages the same.
This is an example study plan, not a completed trial. It contains no measured results, sample-size calculation or uncertainty estimates.
What to agree before the test.
- 1
Decide who to include and what would count as useful
Agree who can take part, which sessions count toward the attendance rate, how long to measure and what improvement would be worthwhile. Set limits on cost and staff time before seeing results.
- 2
Choose and record the comparison groups
Decide whether to assign individual participants or whole groups at random. Account for repeat attendance and reminders shared between participants. Agree permissions, privacy safeguards and exceptions before starting.
- 3
Measure the same way in both groups
Record each person’s assigned group, eligible sessions, attendance, failed messages and staff time. Note missing records and other changes, such as new instructors. Agree how cancellations count.
- 4
Compare the groups as assigned
Include people who did not read the reminder in their assigned group. Account for repeat attendance and any group assignment. Once data are available, report an uncertainty interval suited to the study and explain missing records or departures from the plan.
From evidence to action
The result may
remain inconclusive.
Weigh the likely benefit, uncertainty, cost and staff time together. A statistically significant result alone does not tell you whether the reminder is worth using.
A useful improvement at an acceptable cost
Consider using the reminder more widely if the evidence supports the improvement you agreed was worthwhile and the cost is acceptable. Keep checking how it performs.
Too little benefit or too much cost
Keep the usual process or try a different reminder. Be clear whether the evidence suggests little benefit or is too uncertain to tell.
Still too uncertain to decide
Collect more information only if it is worth the cost. A wide uncertainty interval or a result that is not statistically significant does not prove the reminder has no effect.
When a randomized test
isn’t practical.
You may still learn from existing records, but the comparison needs care. Attendance can change because of the season, different participants, a new timetable or who chooses to take part.
Explain what the evidence can and cannot tell you. Predicting demand, recording attendance and finding out what caused a change are different tasks.
How we test the workWhat evidence would change the decision?
Describe how you handle it now and what gets in the way.