Evaluation:

The final stage of the ADDIE model is to evaluate whether the learning project is achieving its intended goals. Evaluation is performed in both formative and summative assessments. Formative assessments are performed at each stage of the Addie process. Summative assessments are performed after the project has been implemented, and provide opportunities to perform objective measurements of goals achieved, as well as providing users opportuniteis to provide feedback via questionnaires.

This page demonstrates the goals of the course, and how achievement of those goals will be evaluated, including a questionnaire for pilot users to provide Kirkpatrick Level 1 feedback on the course.

E-learning Goals

My e-Learning project is intended to be a small part of larger, 18-week long training course for new Adjudicators-in-training. The immediate goals of the e-Learning project itself are that learners will be able to identify common musculoskeletal impairments, will be able to name common signs, symptoms, and lab findings of musculoskeletal impairments, will understand the requirements of SSA’s musculoskeletal listings as they are described in the Preamble to the listings, and that they will understand enough about the actual Musculoskeletal Listings themselves to be able to identify what listing applies to a particular claim.

However, as part of the greater process, of learning how to be an independent and efficient disability adjudicator for Washing DDS, my e-Learning project has the following goals:

• The e-Learning format of this material will allow for less classroom time to be spent explaining the declarative knowledge contained in the listings. As a result, more classroom time will be available to help learners gain procedural knowledge of how to apply the rules and principles of evaluating musculoskeletal claims.
• As a result, learners who are taught using this e-Learning method will complete the 18-week training course with a greater understanding not only of the declarative content, but also of procedural application of training material
• In the long term, trainees will be better and more quickly equipped to serve the people of the State of Washington by independently and efficiently making well-documented Social Security Disability eligibility determinations.

Kirkpatrick Levels

[List what your specific e-learning is evaluating at each level.]

Level 1 Reaction– At level 1, surveys will be given to users and the trainers working directly with learners, evaluating whether the material was accessible, well organized, and focused to learner needs

Level 2 Learning – At level 2, Learning will be assessed via the use of summative assessments of knowledge at the end of the module, as well as a summative final assessment at the end of the 18-week training class. These assessments will measure what the users learned from both the Musculoskeletal module, and the training course as a whole. The summative assessments include opportunities for users to demonstrate both declarative and procedural knowledge.

Level 3 Behavior – Beginning in week 4 of the classroom training program, learners will be assigned cases, and their first cases will be those with musculoskeletal impairments. Each learner also will be assigned to work with a single member of the training staff, who provides mentoring and 1:1 feedback while the individual applies what they have learned to the cases they receive. Observation and feedback during this time will occur on a daily basis. Learners will attempt to demonstrate that they can apply what they have learned to an actual case, and trainers will document their work, and provide feedback and correction. Trainers will be required to maintain logs of question and feedback, learner difficulties and progress during this time, which demonstrates learner progress toward outcomes. The logs will be reviewed by the trainers teaching the course. Successful application of elements of the lesson to musculoskeletal claims will be evidence that this particular module has successfully done its part to achieve intended outcomes. If difficulties are observed, evaluation will be required to determine whether observed issues are individual in nature, or are in fact global training issues, and need to be addressed by an adjustment in instruction and evaluation.

Trainee cases continue to be followed and logged for at least 6 months, and up to 9 months depending on the need for support. Important checkpoints regarding their understanding and application of concepts is logged by both trainers and supervisors during this time, and can be used to assess how well learning outcomes are translating into changed behavior.

Level 4 Results – every 6 weeks post training, statistics demonstrating elements of case work quality and case management (cases pending, cases closed, QA returns, aged cases, etc.) will be pulled. These statistics serve as measures of the individual’s ability to accurately, efficiently, and independently make disability adjudication decisions. These numbers will be compiled and compared against previous classes at the same point in time to determine whether instructional changes have had an effect on results.

Assessment Techniques and Tools

Beginning in week 4 of the classroom training program, learners will be assigned cases, and their first cases will be those with musculoskeletal impairments. Each learner also will be assigned to work with a single member of the training staff, who provides mentoring and 1:1 feedback while the individual applies what they have learned to the cases they receive. Observation and feedback during this time will occur on a daily basis. Learners will attempt to demonstrate that they can apply what they have learned to an actual case, and trainers will document their work, and provide feedback and correction. Trainers will be required to maintain logs of question and feedback, learner difficulties and progress during this time, which demonstrates learner progress toward outcomes.

The logs will be reviewed by the trainers teaching the course. Successful application of elements of the lesson to musculoskeletal claims will be evidence that this particular module has successfully done its part to achieve intended outcomes. If difficulties are observed, evaluation will be required to determine whether observed issues are individual in nature, or are in fact global training issues, and need to be addressed by an adjustment in instruction and evaluation. This is the stage of feedback that is most likely to reflect on the effectiveness of the 2 day module—if individuals are struggling to apply concepts from the Musculoskeletal System module to effectively adjudicate claims with musculoskeletal impairments, this feedback may provide evidence of the need for revision.

In terms of collecting data concerning the entire course, immediately following the 18 week course, a user survey will be given to all learners. It will ask learners to rate how specific elements of the course served their ability to learn, which elements of the course did not serve their ability to learn, and how much they feel they learned in the course as a whole. This feedback will be gathered by the training manager, and given to the training staff to review and consider before the next training class begins. Data indicating that course design did not achieve intended outcomes will be considered and course revision may be implemented.

Finally, every 6 weeks post training, statistics demonstrating elements of case work quality and case management (cases pending, cases closed, QA returns, aged cases, etc.) will be pulled. These statistics serve as measures of the individual’s ability to accurately, efficiently, and independently make disability adjudication decisions. These numbers will be compiled and compared against previous classes at the same point in time to determine whether instructional changes have had an effect on results.

The training staff will hold regular team meetings and an annual in-person conference in which we will review feedback, analyze individual adjudicator performance, and assess adjudicator class performance in comparison to other adjudicator classes. Trainers will use this information to re-design training elements to better achieve intended outcomes.

Questionnaire

• Rate your overall satisfaction with the e-Learning module

1. 1. Very Dissatisfied
   2. Dissatisfied
   3. Neither Satisfied nor dissatisfied
   4. Satisfied
   5. Very satisfied

• Rate your feeling of being engaged with the learning material

1. 1. Very disengaged
   2. Disengaged
   3. Neither engaged nor disengaged
   4. Engaged
   5. Very engaged

• The learning objectives were clearly communicated

1. 1. Strongly disagree
   2. Disagree
   3. Neither Agree nor Disagree
   4. Agree
   5. Strongly Agree

• The material was well organized and easy to follow.

1. 1. Strongly Disagree
   2. Disagree
   3. Neither Agree nor Disagree
   4. Agree
   5. Strongly agree

• The learning module was easy to navigate.
  1. Strongly disagree
  2. Disagree
  3. Neither Agree nor Disagree
  4. Agree
  5. Strongly Agree

• The practice scenarios were relevant to tasks performed by adjudicators
  1. Strongly disagree
  2. Disagree
  3. Neither Agree nor Disagree
  4. Agree
  5. Strongly Agree

• The knowledge check questions were related to the presented material

1. Strongly Disagree

2. Disagree

3. Neither Agree nor Disagree

4. Agree

5. Strongly Agree

• What suggestions do you have for improving this training course for future learners?