5 Evaluation

This e-learning module tackles the crucial first step in navigating a safe and confident online job search: website verification. By mastering these techniques, learners will gain the skills to identify legitimate companies, reducing the risk of online scams. This empowers them to make informed decisions throughout their job search journey.

Level 1: Reaction

To assess learner satisfaction, I will utilize a post-training survey. This survey will incorporate a mix of formats:

• Closed-ended Questions (Likert Scale): These structured questions allow learners to easily express their level of agreement with statements like:
  • “How satisfied were you with the overall quality of the training?”
  • “After completing this course, how prepared are you to use website verification techniques during your job search?”
• Open-ended Questions: These open-ended prompts encourage learners to elaborate on their experiences with the course. For example:
  • “What did you find most valuable about this training on identifying website legitimacy?”
  • “Briefly explain what aspects of the course helped you feel prepared (or unprepared) to use website verification techniques during your job search.”

Level 2: Learning

To assess the impact of the course on learner knowledge and skill development, I will utilize the following approach:

1. Pre- and Post-Training Assessments: These assessments will be administered before and after the course. Comparing results will provide a clear picture of how learners’ knowledge improved upon completion.
2. Integrated Practice Quizzes: Throughout the course, learners will encounter practice quizzes after each lesson. These quizzes serve a dual purpose:
   • Applying Knowledge: They allow learners to immediately apply the concepts they’ve just learned, solidifying their understanding.
   • Final Assessment Preparation: The practice quizzes act as a stepping stone, preparing learners for the comprehensive final assessment.

Level 3: Behavior

A month after completing the course, a follow-up survey will be sent to the learners to gauge their application of the learned skills. This survey will pose two key questions:

• Frequency of Use: “Since completing the course, how often have you integrated website verification techniques into your job search process when evaluating company websites?”
• Putting Skills into Action: “Can you share a specific instance where you used website verification techniques learned in the course to identify a potentially fraudulent company during your job search?”

These questions aim to provide insights into:

• Learner Engagement: How often are learners actively applying the website verification techniques?
• Real-World Application: Can learners successfully apply the learned techniques to identify potentially fraudulent companies in their job search?

Level 4: Results

Since I’m sharing this course with just friends and family, measuring its impact on website verification practices requires a different approach. Here’s how I plan to gauge its effectiveness:

1. Tracking Course Engagement:

• Completion Rates: I’ll keep an eye on the completion rate to see how many friends and family members make it through the entire course.

2. Post-Course Follow-up:

• Informal Interviews: About a month after they finish the course, I’ll reach out to participants for informal interviews. I’ll ask open-ended questions like:
  • “In what ways did the website legitimacy training help you?”
  • “Have you come across any suspicious websites lately? Did the course help you evaluate their legitimacy?”

These interviews will give me valuable insights into:

• Perceived Value: Whether friends and family found the website verification techniques helpful and informative.
• Real-World Application: Any personal stories about using the techniques to identify suspicious websites or feeling more confident online.

Tool: Articulate Rise

Aside from its intuitiveness, I can use Rise to create effective assessments by:

• Using its various question types: Rise offers a diverse range of question formats, including multiple choice, fill-in-the-blank, and matching. This caters to different learning styles and assesses various aspects of learner knowledge.
• Creating scenario-based questions: With Rise, I can come up with questions with a scenario format. This allows me to assess their ability to apply their knowledge and skills in a practical setting (e.g., learners will be asked to identify whether the company website in a job description is legitimate or not).
• Providing real-time feedback: Rise also allows me to provide customized real-time feedback for both correct and incorrect answers

Assessment Types

This course utilizes two assessment types:

• Formative Assessments (Practice Quizzes):  These built-in quizzes allow learners to gauge their understanding and identify areas needing improvement before taking the final assessment.
• Summative Assessment (Final Assessment):  This comprehensive assessment measures overall learner mastery of the course content and website verification skills. Learners who fails to meet the passing score of 80% or higher can retake the assessment if necessary. Learners will also have the option to revisit the lessons before retaking the final assessment.

Knowledge Test Types

The assessments incorporate two knowledge test types:

• Declarative Knowledge: I have included questions to test their understanding of the facts, concepts, and terminology related to website legitimacy. This also helps me identify which concepts learners might be struggling.
  • Sample Question: Among the following, what’s the LEAST likely red flag in a commercial business website domain name?
    • (a) Misspellings in the company name
    • (b) A “.gov” extension
    • (c) A hyphen separating two words (e.g., face-book.com)
    • (d) The company name followed by a number (e.g., movies123.net)
• Procedural Knowledge: These questions assess the practical application of the course material. Learners get to experience real-world scenarios.
  • Sample Question: Leverage various Google Search techniques to investigate the legitimacy of the website listed in the job posting below. Enter “Yes” if it’s the company’s legitimate website. If not, provide the correct website using the format www.xxx.xxx (example: www.nokia.com).

Ensuring Reliable and Valid Assessments

• Quantitative Measures for Reliability: The assessments utilize multiple choice, fill-in-the-blank, and similar question types with clear, pre-defined answers. This approach ensures consistency in grading regardless of the time learners choose to take the course.

• Quantitative Assessments for Validity: This method eliminates subjectivity in grading and ensures reliability and consistency. Also, this type of assessment method allows me to easily measure learner performance and easily identify topics requiring more explanation or practice based on frequent errors. To further ensure validity:
  • Align with Learning Objectives: Questions are mapped to the learning objectives established at the beginning, which will help ensure that the questions will assess what I intend to measure.
  • Expert Review: I also had some of my past colleagues review the module and the assessments included to confirm that my project covered the key website verification techniques outlined in the learning objectives.

Feedback Mechanism

Since this is going to be 100% asynchronous, explanatory feedback is provided in both practice quizzes and the final assessment. This type of feedback clarifies the “why” behind correct and incorrect answers.

To gather valuable feedback and identify areas for improvement before a wider launch, this pilot group will participate in pre-course and post-course surveys. These questionnaires aim to assess their learning outcomes and gather insights into the course’s effectiveness.

Pre-Course Survey

• Demographics (optional): Questions about their age, profession, and online experience level to see if these factors influence their learning outcomes.
• Website Evaluation Experience:
  • “Have you ever used techniques to evaluate website legitimacy before?” (Yes/No)
  • (If yes) “Can you share a specific instance where you tried to verify a website’s legitimacy? What was the outcome?” (Open-ended)
• Confidence Level:
  • “On a scale of 1 (Not Confident) to 5 (Very Confident), how confident are you in your ability to identify a legitimate company website during your online job search?” (1 question)

Post-Course Survey

• Overall Course Experience:
  • “On a scale of 1 (Not Satisfied) to 5 (Very Satisfied), how satisfied were you with the overall quality of the website verification training?” (1 question)
• Learning Outcomes:
  • “Do you feel more confident in evaluating website legitimacy after completing the course?” (Yes/No)
  • (If yes) “Can you describe a specific way you might use website verification techniques in the future (e.g., job search, online shopping)?” (Open-ended)
• Course Content:
  • “What aspects of the course were most helpful in learning website verification techniques?” (Open-ended)
  • “Were there any sections of the course you found confusing or difficult to understand? If so, please provide specific details.” (Open-ended)