PQStat The Fleiss' Kappa coefficient and a test to examine its significance
When to use:
The Fleiss' Kappa coefficient determines the agreement of measurements made by several raters (Fleiss, 1971[61]) and is an extension of Cohen's Kappa, which only allows the agreement of two raters to be examined. Each of the \(n\) randomly selected objects may be rated by a different random set of \(k\) raters. The analysis is based on data transformed into a table with \(n\) rows and \(c\) columns, where \(c\) is the number of possible categories to which raters assign the objects studied.
Assumptions
- measurement on a nominal scale – any ordering of the categories is not taken into account.
Definitions of measures
The Kappa coefficient (\(\hat \kappa\)) – a value of \(\hat \kappa=1\) means full agreement among raters, while \(\hat \kappa=0\) means the level of agreement that would arise if the raters' opinions were given at random. Negative values of Kappa, in turn, indicate agreement lower than the level expected by chance.
How to set up the analysis
Statistics \(\to\) Nonparametric tests \(\to\) Fleiss Kappa.
Performing and interpreting the analysis
Example 1. (temperament.pqs file)
20 volunteers take part in a game aimed at determining their personality type. Each volunteer has ratings given by 7 different observers (most often people from their close circle or family). Each observer was introduced to the basic traits describing temperament for each personality type: choleric, phlegmatic, melancholic, sanguine. We examine the observers' agreement in assigning personality types. An excerpt of the data is shown in the table below.
Significant agreement applies to every rating, as confirmed by the report summarising agreement for each stratum (for each rating) and the graph showing the individual Kappa coefficients along with the overall summary Kappa.
It may be interesting to note that the highest agreement concerns the rating of phlegmatics (Kappa=0.479952). With a small number of people observed, it is also worth plotting a graph showing how the observers rated each of them.
In this case, only person no. 14 received an unambiguous personality-type rating – sanguine. Persons no. 13 and 16 were rated as phlegmatic by 6 observers (out of 7 possible). For the remaining persons, agreement among the ratings was somewhat lower. The personality type that seems hardest to define characterises the last person, who received the most varied set of ratings.
The Kappa coefficient (\(\hat \kappa\)) is expressed by the formula:
where:
\(P_o=\frac{1}{kn(k-1)}\sum_{i=1}^n\sum_{j=1}^c x_{ij}-kn\),
\(P_e=\sum_{i=1}^c q_j^2\),
\(q_j=\frac{1}{km}\sum_{i=1}^n x_{ij}\),
\(x_{ij}\) – the number of raters assigning object \(i\) to category \(j\).
For the coefficient \(\hat \kappa\), a standard error \(SE\) can be determined, which allows its statistical significance to be tested and asymptotic confidence intervals to be calculated.
The Z-test for the significance of the Fleiss' Kappa coefficient (\(\hat \kappa\)) (Fleiss, 2003[63]) is used to verify the hypothesis about the agreement of the ratings of several raters, and is based on the coefficient \(\hat \kappa\) calculated for the sample.
Hypotheses:
\(\begin{array}{cl} \mathcal{H}_0: & \kappa= 0, \\ \mathcal{H}_1: & \kappa \ne 0. \end{array}\)
Formulas: The test statistic is defined by:
The \(Z\) statistic asymptotically (for large sample sizes) has the normal distribution.
The \(p\) value, determined on the basis of the test statistic, is compared with the significance level \(\alpha\): if \(p \le \alpha\), we reject \(\mathcal{H}_0\) in favor of \(\mathcal{H}_1\); if \(p > \alpha\), there is no reason to reject \(\mathcal{H}_0\).