MODULE 16 · Skill training · 110 MINUTES

Program evaluation, outcome reporting, and operational improvement

Learn the principle. Practice the work. Preserve what you learn.

Your learning goals

Define measures and denominators before interpreting results.
Analyze the supplied synthetic dataset accurately.
Turn evidence into a focused operational improvement.

Preparation

Measure the intended learning or service outcome

Begin with the objective and decide what evidence could reasonably indicate progress. Define each measure, the eligible population, collection point, denominator, and interpretation limit. Participation counts, resource completion, perceived clarity, demonstrated planning quality, and follow-up action are different measures. Keep them separate. A feedback response rate uses feedback responses divided by the eligible group you define; a favorable-response percentage uses the respondents who answered the relevant item. Missing responses remain missing. An evaluation instrument should ask only what is useful for the program decision and should follow the current consent, privacy, and data-handling arrangements. These training calculations use entirely synthetic records.

Analyze the synthetic data carefully

The supplied dataset contains 12 fictional enrollees. Ten attended, eight provided paired pre/post confidence scores and feedback, and six provided follow-up action information. Attendance is 10/12, or 83.3%. Feedback among attendees is 8/10, or 80.0%. The eight paired baseline scores sum to 19 and average 2.375; the paired post scores sum to 29 and average 3.625. The mean within-person change is 1.25 points on the illustrative 1–5 scale. Four of six follow-up respondents report completing their chosen action, or 66.7%; the observed completion count is four of ten attendees, or 40.0%. Those two percentages answer different questions. The dataset cannot establish a causal effect or represent actual Q45 performance.

Report limitations and improve the process

Write a concise interpretation that names the sample, denominator, missing data, measurement type, and follow-up timing. A self-reported confidence change is evidence about the reported score, with uncertainty about other outcomes. Examine practical patterns before recommending a change. In the synthetic example, two feedback respondents request more guidance, so an additional worked example is a plausible next revision. Record the change, owner, version, and measure that will evaluate the revision. Keep a small improvement register connected to the existing project tracker. Distinguish completed outputs, measured outcomes, and longer-term aspirations so the program can learn honestly and decide where to invest effort.

Worked example

title

Worked example: a defensible synthetic evaluation summary

body

“Among 12 fictional enrollees, ten attended. Eight attendees completed paired ratings; their mean confidence score increased from 2.375 to 3.625 on an illustrative 1–5 scale. Four of six follow-up respondents reported completing an action. Missing follow-up information limits interpretation for all attendees. These synthetic data support practice in calculation and reporting. A proposed improvement adds a second worked example and evaluates whether clarity ratings improve in a future authorized pilot.” The summary preserves both the observed changes and their limits.

Your practical assignment

Produce a synthetic evaluation and improvement brief

Inspect the dataset and data dictionary supplied with this curriculum.
Calculate enrollment, attendance, feedback, paired scores, mean change, follow-up response, and action completion using explicit denominators.
Create an objective-to-measure table and a missing-data summary.
Write a 700–1,000-word evaluation brief with limitations.
Recommend one resource improvement and one operational improvement, each with an owner role, evidence, and next measure.

Evidence to prepare

A calculation table with formulas and denominators.
A 700–1,000-word synthetic evaluation brief.
A two-item improvement register.

Assignment scoring guide

Use these criteria for self assessment and feedback. Each criterion contributes 25 points to the 100-point assignment rubric.

Criterion	Points	Evidence
Calculation accuracy	25	Counts, means, percentages, and paired calculations are correct.
Measurement clarity	25	Measures, denominators, missingness, and self-report limits are explicit.
Interpretation	25	Conclusions remain proportionate to the synthetic evidence.
Actionable improvement	25	Recommendations connect a finding to a specific change and subsequent measure.

Reflection

Which denominator would a reader be most likely to misunderstand?
What additional evidence would help assess whether the proposed improvement worked?

Scoring and revision guidance

Evidence is absent or cannot be assessed.

Limited evidence; substantial guidance and revision are needed.

Partially demonstrated; a material element needs revision.

The essential requirement is met; minor corrections remain.

Complete, accurate, traceable, and usable evidence for the criterion.

Review the explanation for each missed question, revisit the relevant lesson, and retry. Revise practical work against the rubric and document the change.