Validity Data Analysis

Compute content validity ratio for test items.Purpose

• Interpret a validity coefficient, calculate the coefficient of determination, and conduct a test of significance for a validity coefficient.
• Examine the convergent and discriminant evidence of validity.

In this assignment, you will compute and interpret the validity evidence. In Part 1, you will calculate the content validity ratios. In Part 2, you will compute a validity coefficient using Jamovi. In Part 3, you will conduct a correlation analysis in Jamovi to obtain the convergent and discriminant evidence of test validity.

Please submit your work on Blackboard.

You can either:

1. Type in your work and answers in MS Word version of the workbook, save and submit.

or

1. Print out the workbook, write down your work and answers, and scan the completed workbook, and submit.

Background

You are working as a talent management intern for the multi-state grocery chain. You are developing a 10-item customer orientation measure to select grocery store cashiers.

How to Complete

Part 1: Content Validity Ratio (20 points)

In Part 1, you will compute the content validity ratio to summarize the subject matter experts’ judgment on each test item on the customer orientation scale.

You have asked 14 SMEs (grocery store managers, experienced cashers) to rate each item if it is “essential,” “useful,” or “not necessary” for promptness as a cashier.

1. What is the content validity ratio for each item (Item 1–Item 10)?
2. What minimum value of content validity ratio is expected in this case? (Hint: Check Lawshe’s table on the slide presentation!) Based on the content validity ratio of 10 items, would you suggest any items to be revised or removed?

Part 2: Validity Coefficients (20 points)

There are step-by-step instructions as well as a video on “How to get validity coefficient in Jamovi” in the slide presentation.

In Part 2, you are examining the validity evidence based on the relationship to external variables. You decided to use a concurrent design. You asked 100 current cashiers to complete the new customer orientation scale and obtained their most recent job performance score from the personnel file.

Open Jamovi data file “PSY590_jobperformance_FA20.omv”.

1. What is a criterion in this case?

Run a correlation analysis and answer Questions 4 through 9.

1. What is the validity coefficient of the customer orientation score predicting job performance for cashiers?
2. Does the customer orientation score predict cashiers’ job performance? Support your answer with a correlation/validity coefficient (r) and its p-value.
3. How much variance in job performance is explained by the customer orientation scores? Report the coefficient of determination and its interpretation.

Part 3: Discriminant and Convergent Validity (20 points)

In Part 3, you will continue examining the validity evidence based on the relationship to external variables. In addition to the customer orientation scale, you asked the 100 cashiers to complete two additional scales: Team-orientation scale and service competency scale.

Both customer orientation and team-orientation scales are used for selecting cashiers. While both are deemed critical for cashiers to perform their job successfully, these two scales measure different characteristics.

The service competency scale is a commercially available selection measure for customer service jobs. This scale measures the job applicants’ characteristics that are similar to the new customer orientation scale.

Please use the same data file from Part 2 (Jamovi data file “PSY590_jobperformance_FA20.omv”).

Run a correlation analysis and answer Questions 7 and 8.

1. Is there evidence of convergent validity for the customer orientation scale? Please support your answer with an appropriate correlation coefficient (r) and its p-value.
2. Is there evidence of discriminant validity for the customer orientation scale? Please support your answer with an appropriate correlation coefficient (r) and its p-value.