Inter-Rater Reliability
By definition, inter-rater is the statistical measurement that tends to determine the similarity between the data that is collected by the different raters. Raters are those who score or measure the behavior, performance or skill in an animal or human (Sackett, 2006). The raters include various interviewers, psychologists that measure the number of times that a subject reacts in a particular manner. Importantly, the raters are often expected to come up with observations that do not vary much. Notably, sometimes there may be sources that bring disagreements between the inter-raters. For instance, the measurements that usually involve the quality of judgment that is subjective like a speaker's presentation skills, witness evaluation of the jury's credibility, as well as bedside manners of a physician, may cause disagreements in the inter-rater reliability (DeVellis, 2016).
Sources of Variance Error
The variation that exists across the raters in various procedures of measurement and the variability in the measurement interpretation results are some of the sources of variance error in measurement rating (Kline, 2005). Therefore, in the challenging or ambiguous scenarios, the raters follow specific guidelines for reliability purposes. There is usually an increase in experimenter's bias in the ratings that are not guided by these guidelines. Repeated measurement processes can be rectified through retraining of the raters for a period to instill the instructions as well as measurement in the raters and avoid rater drift. On the flip side, operational rater agreement liability includes the agreement in the official performance rating by the reliable raters, these raters' agreement about ratings that ought to be awarded, as well as their agreement on the performances levels of goodness (DeVellis, 2016). The statistics include the kappa, joint probability, correlation coefficient, limits agreement, and intra-class correlation coefficient.
References
DeVellis, R. F. (2016). Scale development: Theory and applications (Vol. 26). Sage publications
Sackett. (2006). High-Level Mobility Assessment Tool (HiMAT): Interrater Reliability, Retest Reliability, and Internal Consistency. Physical Therapy. doi: 10.1093/ptj/86.3.395
Kline, T. (2005). Psychological testing: A practical approach to design and evaluation. Thousand Oaks, CA: Sage Publications.
