This Evaluation section reflects the Evaluation Phase of ADDIE, focusing on how the effectiveness of the training will be measured across multiple levels. Using Kirkpatrick’s Four Levels of Evaluation, this section outlines how learner reaction, knowledge acquisition, on-the-job behavior change, and organizational results will be assessed. It also details the assessment strategies, tools, and rationale used to ensure the course delivers measurable and meaningful outcomes. By combining formative and summative assessments, observation tools, and data-driven evaluation methods, this section demonstrates a comprehensive approach to ensuring the course not only meets its learning objectives but also advances broader organizational goals related to patient safety and care quality.

E-learning Goals

The primary goal of this course is to prevent patients with life-threatening symptoms from receiving delayed care and reduce the number of delayed care incidences reported on the Risk Management Report.

Kirkpatrick Levels

Level 1 Reaction

Post-Course Survey

Level 1 evaluation will focus on learner engagement and satisfaction with the course design, content, and delivery will be assessed using a post-course survey.

Level 2 Learning

Level 2 evaluation will consist of a combination of formative and summative assessment.

Formative Assessments

Each lesson within the course will have a knowledge check that must be completed to progress to the next section. Feedback will direct participants who answer incorrectly where to go back in the course for further review of the material.

Summative Assessment

Participants will complete a scenario-based assessment at the end of the course. They will need to score 100% to receive a certificate of completion due to the high-stakes nature of the content covered in the course. There will be opportunities for users to retake the assessment, if they do not initially pass, after reviewing the course material.

Level 3 Behavior

Observation Checklist

Level 3 evaluation will look at the following areas of employee on the job performance assessed using the Observation Checklist by the Medical Director: proper triage judgment during patient check-in. Adherence to emergency protocols when symptoms of life-threatening conditions are present. Accuracy and consistency in notifying qualified medical staff.

Level 4 Results

Risk Management Data

Level 4 evaluation will look at the following areas of the clinic’s overall goal of providing prompt, safe, and accurate patient care. These areas will be assessed using data already collected by the medical director.

Assessment Techniques and Tools

Types of Knowledge

The course assessments will be designed to assess both procedural and declarative knowledge. The course’s formative assessments will focus primarily on declarative knowledge and the course’s summative assessment will focus primarily on procedural knowledge. The course’s declarative knowledge will include recognizing symptoms of life-threatening symptoms, recalling notification procedures and emergency protocols. The procedural knowledge will include making judgment calls given a patient’s symptoms, deciding when medical staff need to be notified for a patient’s potential life-threatening symptoms, and making a judgement call when registration can continue as normal when a patient does not have potentially life-threatening symptoms.

Formative Assessments

Each lesson within the course will have a knowledge check that must be completed to progress to the next section. Feedback will direct participants who answer incorrectly where to go back in the course for further review of the material.

Summative Assessment

Participants will complete a scenario-based assessment at the end of the course. They will need to score 100% to receive a certificate of completion due to the high-stakes nature of the content covered in the course. There will be opportunities for users to retake the assessment, if they do not initially pass, after reviewing the course material.

Instructor Role/Feedback Rational

This course is designed to be used any time a new medical receptionist is hired. It will be asynchronous, always available, and self-paced. There will be no real-time instructor involved in the delivery of the course. Feedback will be included with each response in the knowledge checks and the scenario-based assessment.

Data Collected

Quantitative data will be collected during the knowledge checks and the final assessment. Qualitative data will not be collected due to the need for the course to be asynchronous and readily available anytime a new medical receptionist is hired.

Assessment Grading

For liability purposes each medical receptionist will need to receive a score of 100% to receive a certificate of completion, which will be submitted to the medical director of the facility.

The assessment will be tightly aligned to the learning objectives of the course to ensure reliability. This alignment helps guarantee that the test consistently measures the intended knowledge and skills across different participants and over time. A follow-up assessment will be administered three months after the initial training. This delayed assessment will evaluate knowledge retention and confirm that the original assessment accurately measured the course content and long-term understanding, further supporting its validity.

Assessment Tools

Storyline 360 allows for the development of highly interactive and responsive assessments that are seamlessly integrated into the course modules. It supports automated feedback, branching logic, and tracking of learner progress, making it ideal for a course that requires learners to master content independently and demonstrate readiness before engaging with real patients.

Description of Tools Used and Rationale

Formative Assessments
The course’s formative assessment knowledge checks will be built in Storyline 360. They will include:

Multiple choice questions that test recognition of symptoms and emergency protocol steps. Matching activities to assess appropriate steps to be taken when an emergency notification needs to be given to medical staff in the treatment area. A Pick Many question will be used to assess who qualified medical staff members are in the facility. Feedback will be embedded that guides learners to revisit relevant content if they answer incorrectly.

These question types are effective for efficiently gauging recall and understanding in an asynchronous digital environment where real-time instructor feedback is not available.

Summative Assessment
The scenario-based assessment will also be built in Storyline. It will require learners to: Review simulated patient encounters with presented symptoms. Make critical decisions, such as whether to notify medical staff or continue registration. Locating qualified staff members to notify and following the clinic’s emergency protocols.

Given the life-and-death stakes of urgent care triage, a traditional multiple-choice test would not sufficiently assess whether learners can apply knowledge under pressure. Scenario-based simulations provide a more authentic, performance-based measure of readiness.

Course Pilot User Questionnaire

General Experience

1. On a scale of 1–5, how easy was it to navigate the course?

2. Were the instructions clear at each stage of the course? If not, where did you encounter confusion?

3. Did the course load and function properly on your device? If not, please describe the issue.

Content Clarity and Relevance

4. Did the content feel relevant to your day-to-day role as a medical receptionist?

5. Was any part of the course confusing or difficult to understand?

6. Were there any medical terms or procedures that you feel need more explanation?

Engagement and Interactivity

7. How engaging did you find the course? (1–5 scale)

8. Did the knowledge checks and scenario-based assessment feel helpful and realistic?

9. Did you feel the scenarios reflected situations you might actually encounter at work?

Assessment and Feedback

10. Did the feedback provided in the knowledge checks help you confirm your understanding of the material?

11. Did the final scenario-based assessment feel fair and aligned with what was taught?

12. Did the requirement to get 100% feel reasonable for the content?

Overall Usefulness

13. What is one thing you learned in this course that you didn’t know before?

14. Is there anything you feel should be added or removed from the course?

15. Do you feel more confident in making triage decisions after completing this training?