Evaluating Annotation Consistency in Offensive Language Detection: A Data Analytics Approach on the TweetEval Dataset

Abstract

Most machine learning models are not only highly 
dependent on difficult datasets but also on the quality of labeled 
data they are trained on, especially for offensive content detection. 
In this paper, we study the TweetEval dataset to provide a 
comparison of its ground truth with manually annotated labels; 
inter-annotator agreements are applied here as a metric for 
assessing the consistency of annotation. Cohen’s Kappa coefficient 
is used to quantify how much each pair of annotators agreed and 
where they differed. In-depth examination of missed classifications 
demonstrates other difficulties with manual labelling: subjective 
interpretation, context dependency, and annotator bias. The in- 
sights gathered demonstrate how manual annotation can have 
positive and negative effects on further model training practices, 
highlighting the importance of standardized annotation guidelines. 
In their actions, the findings contribute to enhancing offensive 
content detection models by advocating dataset reliability and the 
reduction of inconsistencies in labeling.