2.1 Working definition of cyberbullying

A common starting point for conceptualizing cyberbullying are definitions of traditional (or offline) bullying. Seminal work has been published by Olweus (Reference Olweus1993) and Salmivalli et al. (Reference Salmivalli, Lagerspetz, Björkqvist, Österman and Kaukiainen1996), who describe bullying based on three main criteria, including (i) intention, that is a bully intends to inflict harm on the victim, (ii) repetition, that is bullying acts take place repeatedly over time, and (iii) a power imbalance between the bully and the victim, that is a more powerful bully attacks a less powerful victim. The same criteria are often used to define cyberbullying. Smith et al. (Reference Smith, Mahdavi, Carvalho, Fisher, Russell and Tippett2008), p. 376, for instance, proposed the popular definition of cyberbullying, identifying the phenomenon as “an aggressive, intentional act carried out by a group or individual, using electronic forms of contact, repeatedly and over time, against a victim who cannot easily defend him or herself.”

Other studies, however, have questioned the relevance of the three criteria to define online bullying based on theoretical objections and practical limitations. Firstly, while Olweus (Reference Olweus1993) claims intention to be inherent to bullying, this is hard to ascertain in online conversations, which lack the signals of a face-to-face interaction like intonation, facial expressions, and gestures. In other words, the receiver might get the unintended impression of being offended or ridiculed (Vandebosch and Van Cleemput Reference Vandebosch and Van Cleemput2009). Another criterion that would not hold in online bullying is the power imbalance between bully and victim. In real life, this could mean that the bully is larger, stronger, or older than the victim, but it is difficult to measure online, where it can be related to technological skills, anonymity, or the inability to escape the bullying (Slonje and Smith Reference Slonje and Smith2008). Anonymity and persistence of content are empowering characteristics of the web for the bully: once defamatory or confidential information is posted online, it is hard—if not impossible—to remove. Finally, Dooley and Cross (Reference Dooley and Cross2009) claim that repetition in cyberbullying is problematic to operationalize, as it is hard to estimate the consequences of a single derogatory message on a public page. Even a single act of aggression or humiliation may result in continued distress and humiliation for the victim if it is shared, liked, or read by a large audience (Dooley and Cross Reference Dooley and Cross2009).

The above paragraphs demonstrate that defining cyberbullying is far from trivial, and varying prevalence rates (cf. supra) show that a univocal definition of the phenomenon is still lacking in the literature. Starting from existing conceptualizations, we define cyberbullying as content that is published online by an individual and that is aggressive or hurtful against a victim. The motivation for a rather broad definition of cyberbullying is twofold: first, as mentioned earlier, quantifying repeated offense is difficult, especially in an online environment. Second, context-rich data would allow for a better delineation of cyberbullying, but severe challenges are faced given the sensitivity of the topic and strict general data protection regulations.

Based on this working definition, however, Van Hee et al. (Reference Van Hee, Verhoeven, Lefever, De Pauw, Daelemans and Hoste2015c) developed a fine-grained annotation scheme to signal textual characteristics of cyberbullying that can be considered specific forms of bullying and identify different participant roles in cyberbullying.

2.2 Participant roles in bullying situations

In order to capture the complex social interactions involved in bullying events, we set out to automatically detect and classify bullying utterances as belonging to a specific participant role in the bullying interaction. Participant roles conceptualize typical behavior patterns in bullying situations as social roles. Salmivalli (Reference Salmivalli1999) were among the first to define bullying in these terms. They distinguish six roles: the victims (who are the target of repeated harassment), the bullies (who are the initiative-taking, active perpetrators), the assistants of the bully (who encourage the bullying), the reinforcers of the bully (who reinforce the bullying), the defenders (who comfort the victim, take his/her side, or try to stop the bullying), and the outsiders (who distance themselves from the situation). Their seminal work on bullying relied on surveys with children at schools in real-life bullying situations, but the conceptual framework and sociometrics used in these are later applied to cyberbullying (Salmivalli and Pöyhönen Reference Salmivalli and Pöyhönen2012).

Although traditional studies on bullying have mainly concentrated on bullies and victims, the importance of bystanders in a bullying episode has also been acknowledged (Salmivalli Reference Salmivalli2010; Bastiaensens et al. Reference Bastiaensens, Vandebosch, Poels, Van Cleemput, DeSmet and De Bourdeaudhuij2014). When it comes to prevention many people can take an active role to intervene, especially bystanders. Bystanders can support the victim and mitigate the negative effects caused by the bullying (Salmivalli Reference Salmivalli2010), especially on SNSs, where they have shown to hold higher intentions to help the victim than in real-life conversations (Bastiaensens et al. Reference Bastiaensens, Vandebosch, Poels, Van Cleemput, DeSmet and De Bourdeaudhuij2015). While Salmivalli et al. (Reference Salmivalli, Lagerspetz, Björkqvist, Österman and Kaukiainen1996) distinguish four bystanders, Vandebosch et al. (Reference Vandebosch, Van Cleemput, Mortelmans and Walrave2006) identify three main types, namely bystanders who (i) participate in the bullying, (ii) help or support the victim, and (iii) ignore the bullying.

The lack of social context in near-anonymous online interactions entails a different operationalization of author participant roles: anonymity makes it challenging to link roles to actual persons. It is impossible to ascertain the frequency and different types of roles a person takes as social behavior. The conversational structure in our dataset is simple compared to full dialogue due to the limitations of the ASKfm platform (http://ask.fm/) in which a dialogue consists of only two utterances: a question and a response (which is optional and therefore often missing). Note that users do hold longer continuous conversations by posting several question–answer pairs, but reconstructing a dialog is impossible because of anonymous posting on the platform. Due to limited dialog information, we do not rely on conversational structure but on purely textual utterances that characterize typical social roles in cyberbullying. This allows for social role detection even if little context is available and user information is anonymized.

2.3 Automated detection and analysis of cyberbullying

Although some studies have investigated rule-based approaches (Reynolds, Kontostathis, and Edwards Reference Reynolds, Kontostathis and Edwards2011), the dominant approach to cyberbullying detection involves machine learning, mostly based on supervised (Dinakar, Reichart, and Lieberman Reference Dinakar, Reichart and Lieberman2011; Dadvar Reference Dadvar2014) or semi-supervised learning (Nahar et al. Reference Nahar, Al-Maskari, Li, Pang, Wang and Sharaf2014). The former constructs a classifier using labeled training data, whereas semi-supervised approaches rely on classifiers that are built from a small set of labeled and a large set of unlabeled instances. As cyberbullying detection essentially involves distinguishing bullying from “not bullying” posts, the problem is generally approached as a binary classification task.

A key challenge in cyberbullying research is the availability of suitable data. In recent years, only a few datasets have become publicly available for this task, such as the training sets provided by the CAW 2.0 workshop (Yin et al. Reference Yin, Xue, Hong, Davison, Kontostathis and Edwards2009), the Twitter Bullying Traces dataset (Sui Reference Sui2015), and more recently the Polish cyberbullying corpus that is made available in the framework of PolEval 2019.Footnote ^a Most studies have, therefore, constructed their own corpus from platforms that are prone to bullying content, such as YouTube (Dinakar et al. Reference Dinakar, Reichart and Lieberman2011), Formspring.com (Reynolds et al. Reference Reynolds, Kontostathis and Edwards2011), ASKfm (Van Hee et al. Reference Van Hee, Lefever, Verhoeven, Mennes, Desmet, De Pauw, Daelemans and Hoste2015b),Footnote ^b Instagram (Hosseinmardi et al. Reference Hosseinmardi, Rafiq, Han, Lv and Mishra2016), and WhatsApp (Sprugnoli et al. Reference Sprugnoli, Menini, Tonelli, Oncini and Piras2018). To overcome the problem of limited data accessibility, some studies use simulated cyberbullying data obtained from carefully setup experiments (Van Hee et al. Reference Van Hee, Lefever, Verhoeven, Mennes, Desmet, De Pauw, Daelemans and Hoste2015b; Sprugnoli et al. Reference Sprugnoli, Menini, Tonelli, Oncini and Piras2018). Furthermore, corpora were also compiled covering related subtasks such as aggressive language, offensive language, hate speech, and other abusive language detection. Many of these datasets were compiled in the framework of shared tasks, such as the TRAC shared task on aggression identification (Kumar et al. Reference Kumar, Ojha, Malmasi and Zampieri2018), the HatEval shared task on multilingual detection of hate speech against immigrants and women on Twitter (Basile et al. Reference Basile, Bosco, Fersini, Nozza, Patti, Pardo, Rosso and Sanguinetti2019), and the OffensEval shared task on offensive language identification (Zampieri et al. Reference Zampieri, Malmasi, Nakov, Rosenthal, Farra and Kumar2019b). The latter task is based on the Offensive Language Identification Dataset (OLID) (Zampieri et al. Reference Zampieri, Malmasi, Nakov, Rosenthal, Farra and Kumar2019a), which also includes the coarse-grained two-class annotation of the target of the offensive language (individual vs. group). Insults and threats targeted at individuals are often defined as cyberbulling.

Among the first studies on cyberbullying detection are Yin et al. (Reference Yin, Xue, Hong, Davison, Kontostathis and Edwards2009), Reynolds et al. (Reference Reynolds, Kontostathis and Edwards2011), Dinakar et al. (Reference Dinakar, Reichart and Lieberman2011), who explored the predictive power of n-grams, part-of-speech information (e.g., first and second pronouns), and dictionary-based (i.e., profanity lexicons) information for this task. Similar features were also exploited for the detection of fine-grained cyberbullying categories (Van Hee et al. Reference Van Hee, Lefever, Verhoeven, Mennes, Desmet, De Pauw, Daelemans and Hoste2015b). Studies have also demonstrated the benefits of combining such content-based features with user-based information including previous posts, the user’s age, gender, location, number of friends and followers, and so on (Dadvar Reference Dadvar2014; Nahar et al. Reference Nahar, Al-Maskari, Li, Pang, Wang and Sharaf2014; Al-Garadi, Varathan, and Ravana Reference Al-Garadi, Varathan and Ravana2016; Chatzakou et al. Reference Chatzakou, Kourtellis, Blackburn, De Cristofaro, Stringhini and Vakali2017). Recently, deep neural network (DNN) models have also been applied to cyberbullying (Zhang et al. Reference Zhang, Tong, Vishwamitra, Whittaker, Mazer, Kowalski, Hu, Luo, Macbeth and Dillon2016; Zhao and Mao Reference Zhao and Mao2017; Agrawal and Awekar Reference Agrawal and Awekar2018); in case of comparison with traditional machine learning methods in benchmarking tasks, they currently seem to slightly outperform or perform on par with classifiers like support vector machine (SVM) and even random forest and logistic regression (LR) (Zhang et al. Reference Zhang, Tong, Vishwamitra, Whittaker, Mazer, Kowalski, Hu, Luo, Macbeth and Dillon2016; Kumar et al. Reference Kumar, Ojha, Malmasi and Zampieri2018; Emmery et al. Reference Emmery, Verhoeven, De Pauw, Jacobs, Van Hee, Lefever, Desmet, Hoste and Daelemans2019; Basile et al. Reference Basile, Bosco, Fersini, Nozza, Patti, Pardo, Rosso and Sanguinetti2019). The current state of the art in deep-learning text classification is obtained by large-scale PLMs using the transformer architecture such as BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019) and its derivatives. For offensive language classification in OffensEval (Zampieri et al. Reference Zampieri, Malmasi, Nakov, Rosenthal, Farra and Kumar2019b), BERT-based approaches obtained best performance out of all deep-learning techniques.

The previously discussed studies have in common that they are aimed at detecting cyberbullying or merely abusive language, without processing information about the participant roles in a cyberbullying event. We discuss the only two studies—to our knowledge—in modeling some form of participant roles:

Xu et al. (Reference Xu, Jun, Zhu and Bellmore2012) introduced bully role labeling as a challenge for the field of natural language processing when laying out future paths for research on bullying. The authors experimented with tweets containing keywords like “bully” and “bullied” as bullying traces. They studied traces of real-life bully events, which lead them to define five participant roles: bully, victim, reporter (who reports a bully event on social media), accuser (who accuses someone of bullying), and other. They performed supervised classification on 648 role-annotated tweets. A linear-kernel SVM trained on token uni- and bigrams performed best, yielding a cross-validated accuracy of 61% (no F-score or ROC-AUC scores were given). Furthermore, they experimented with token-level labeling of participant roles where a Conditional Random Field classifier (${F_1}$= 0.47) outperformed an SVM classifier (${F_1}$= 0.36). Their dataset is small compared to ours and is composed using keyword searches, thus limiting the number of relevant posts (i.e., less implicit cyberbullying) and excluding a lot of negative data as well, which biases the dataset (Cheng and Wicks Reference Cheng and Wicks2014; González-Bailón et al. Reference González-Bailón, Wang, Rivero, Borge-Holthoefer and Moreno2014). Because our corpus is randomly crawled instead of collected by keyword search, it contains a more realistic distribution of cyberbullying posts. Moreover, all posts were manually annotated with fine-grained information on cyberbullying types and participant roles, which allowed to identify implicit forms of cyberbullying that would not appear in keyword search.

Raisi and Huang (Reference Raisi and Huang2018) defined a weakly supervised n-gram-based model which includes the sender’s bullying and receiver’s victim scores as parameters in optimization. Their algorithm optimizes parameters for all users and n-gram features that characterize the tendency of each user to send and receive harassing keyphrases as well as the tendency of a keyphrase to indicate harassment. They use a human-curated list of keyphrases as indicators of cyberbullying for weak supervision. This requires that the user profiles are known and available. The authors evaluate their approach on three datasets consisting of Twitter, ASKfm, and Instagram. Anonymous user interactions in their ASKfm corpus were discarded, since they essentially use a communicative sender–receiver model as a proxy for bully and victim participant roles.

In the present research, cyberbullying is considered a complex phenomenon consisting of several types of harmful behavior and realized by different participant roles (Van Hee et al. Reference Van Hee, Verhoeven, Lefever, De Pauw, Daelemans and Hoste2015c). In our previous work on binary and fine-grained cyberbullying detection (Van Hee et al. Reference Van Hee, Lefever, Verhoeven, Mennes, Desmet, De Pauw, Daelemans and Hoste2015b; Reference Van Hee, Jacobs, Emmery, Desmet, Lefever, Verhoeven, De Pauw, Daelemans and Hoste2018), we did not differentiate between participant roles in cyberbullying. To our knowledge, no previous study has been done that aims to automatically predict the participation role in a cyberbullying event on a representative corpus (i.e., containing real-world data and not biased by keyword search).

3.1 Data collection

A Dutch and English corpus were constructed by collecting data from the SNS ASKfm, where users can create profiles and ask or answer questions, with the option of doing so anonymously. ASKfm data typically consist of question–answer pairs published on a user’s profile. The data were retrieved by crawling a number of seed profiles using the GNU Wget software in April and October 2013. The terms of service of ASKfm did not forbid automated crawling at that time. Question–answer pairs were kept together and pairs occurring on the same user page were presented as a conversational thread for annotation as to provide the annotators with as much context as possible. Basic cleaning was applied to the corpus (e.g., removal of non-ASCII characters, multiple white spaces, duplicate and mass spam posts), as well as language filtering. Although the seed profiles were chosen to be of users with Dutch and English as mother tongue, the crawled corpora both contained a fair amount of non-Dutch and non-English data. Non-English and non-Dutch posts were removed, which resulted in 113,698 and 78,387 posts for English and Dutch, respectively.

For more detail on the annotation scheme and dataset, we refer to Van Hee et al. (Reference Van Hee, Lefever, Verhoeven, Mennes, Desmet, De Pauw, Daelemans and Hoste2015b).

3.2 Data annotation

In the following paragraphs, we present our data annotation guidelines as described in Van Hee et al. (Reference Van Hee, Verhoeven, Lefever, De Pauw, Daelemans and Hoste2015c) and focus on different types and roles related to the phenomenon.

3.2.2 Participant roles in cyberbullying

In formulating an annotation scheme for supervised classification of participant roles, annotators were asked to infer the bullying role as an illocutionary act by means of the information present in the ASKfm corpus. The “outsider” from Salmivalli’s participant classification (Salmivalli, Voeten, and Poskiparta Reference Salmivalli, Voeten and Poskiparta2011b) has been left out, given that passive bystanders are impossible to recognize in online text because they do not leave traces. Concretely, four cyberbullying roles were annotated in both corpora:

Harasser: person who initiates the harassment, that is the bully.

Victim: person who is harassed.

Bystander-assistant: person who does not initiate but takes part in the actions of the harasser.

Bystander-defender: person who helps the victim and discourages the harasser from continuing.

The annotation scheme describes two levels of annotation. Firstly, the annotators were asked to indicate, at the post level, whether the post under investigation contained traces of cyberbullying. If so, the annotators identified the author’s role as one out of the four mentioned above. Secondly, at the subsentence level, the annotators were tasked with the identification of a number of fine-grained categories related to cyberbullying. More concretely, they identified all text spans corresponding to one of the categories described in the annotation scheme. To provide the annotators with some context, all posts were presented within their original conversation when possible. All annotations were done using the Brat rapid annotation tool (Stenetorp et al. Reference Stenetorp, Pyysalo, Topić, Ohta, Ananiadou and Tsujii2012). Figure 1 shows example annotations of participant roles.

Figure 1.

Examples of ASKfm posts including “harasser,” “victim,” “bystander-defender,” and “bystander-assistant” instances in the Brat annotation tool.

Because of the rare occurrence of bystander-assistants in the dataset (English n = 24 (0.45%), Dutch n = 38 (0.74%) cf. Table 1), this class was merged with the harasser class for the machine learning experiments. First, classification difficulty increases significantly with data imbalance. Given the low occurrence of bystander-assistant instances, there was too little data to learn a discriminative model for these posts. Second, the textual content of the post assigned to both classes is similar; it is from the (incomplete) context that the initiation nature of harasser posts was determined by annotators. Note that the four labels will be maintained in the discussion of role distribution and dataset statistics presented in the next section.

Table 1.

Distribution of participant roles in the English and Dutch cyberbullying corpus

3.3 Dataset and annotation statistics

The English and Dutch corpora were independently annotated for cyberbullying by trained linguists after supervised instruction and practice with the guidelines. All were Dutch native speakers or English second-language speakers. To demonstrate the validity of our annotations, IAA scores were calculated using Kappa (${\kappa}$) on a subset of the English and Dutch ASKfm corpus. For English, 3882 posts were annotated by four raters and for Dutch 6498 posts were annotated by two raters. For both IAA-corpora, all posts were annotated by each rater to have full overlap. Inter-rater agreement for Dutch is calculated using Cohen’s Kappa (Cohen Reference Cohen1960), while Fleiss’ Kappa (Fleiss Reference Fleiss1971) is used for the English corpus due to there being more than two raters. Kappa scores for the identification of cyberbullying are ${\kappa=0.69}$ (Dutch) and ${\kappa=0.59}$ (English), which point to substantial and moderate agreement (Landis and Koch Reference Landis and Koch1977). Kappa scores for participant roles are ${\kappa=0.65}$ (Dutch) and ${\kappa=0.57}$ (English), again pointing to respectively substantial and moderate agreement. We also computed cross-averaged ${F_1}$-score of the annotators: Dutch annotations have a 0.63 and English annotations a 0.59 macro-averaged ${F_1}$-score.

As the corpus consists of a random crawl of the ASKfm website, we have at our disposal a realistic dataset with regard to the occurrence of cyberbullying traces, which has been annotated at a fine-grained level. This stands in contrast with many previous studies that use keyword-based search to collect bully traces. The English and Dutch corpus contain 113,694 and 78,387 posts, respectively (cf. Table 2). A similar skewed distribution of bullying versus not bullying posts can be observed in both languages, as well as a comparable distribution of bullying roles.

Table 2.

Statistics of the English and Dutch cyberbullying corpus

4.1 Text preprocessing and feature engineering

As preprocessing, we applied tokenization, part-of-speech-tagging, and lemmatization to the data using the LeTs Preprocess Toolkit (van de Kauter et al. Reference van de Kauter, Coorman, Lefever, Desmet, Macken and Hoste2013). In supervised learning, a machine learning algorithm takes a set of training instances (the label of which is known) and seeks to build a model that generates a desired prediction for an unseen instance. To enable the model construction, all instances are represented as a vector of features (i.e., inherent characteristics of the data) that contain information that is potentially useful to distinguish bullying from not bullying content.

We experimentally tested whether cyberbullying events can be automatically recognized by lexical markers in a post. To this end, all posts were represented by a number of information sources (or features) including lexical features like bags of words, sentiment lexicon features, and topic model features, which are described in more detail below. Prior to feature extraction, some data cleaning steps were executed, such as the replacement of hyperlinks and @-replies, removal of superfluous white spaces, and the replacement of abbreviations by their full form (based on the chatslang.com lexicon). Additionally, tokenization was applied before n-gram extraction and lemmatization for sentiment lexicon matching, and stemming was applied prior to extracting topic model features.

After preprocessing the corpus, the following feature types were extracted:

Word n-gram bags of words: binary features indicating the presence of word unigrams, bigrams, and trigrams.

Character n-gram bags of words: binary features indicating the presence of character bigrams, trigrams, and fourgrams (without crossing word boundaries). Character n-grams provide some abstraction from the word level and provide robustness to the spelling variation that characterizes social media data.

Term lists: one binary feature derived for each one out of six lists, indicating the presence of a term from the list in a post: proper names, “allness” indicators (e.g., always, everybody), diminishers (e.g., slightly, relatively), intensifiers (e.g., absolutely, amazingly), negation words, and aggressive language and profanity words. We hypothesize that “allness” indicators, negation words, and intensifiers are informative of often hyperbolic bullying style. Diminishers, intensifiers, and negation words were all obtained from an English grammar describing these lexical classes or existing sentiment lexicons (see further). Person alternation is a binary feature indicating whether the combination of the first and second person pronoun occurs in order to capture interpersonal intent. We hypothesize posts that contain person alternation and proper names to be informative because cyberbullying is a directed and interpersonal communicative act.

Subjectivity lexicon features: Subjectivity lexicons provide categorical or continuous values for the affective connotation of a word. Sentiment can correspond to sentiment polarity, that is positive, negative, or neutral, or as more fine-grained categories of emotions or psychological processes (as is the case for Linguistic Inquiry and Word Count (LIWC) Pennebaker et al. Reference Pennebaker, Francis and Booth2001). From the sentiment lexicons, we derive positive and negative opinion word ratios, as well as the overall post polarity. For Dutch, we made use of the Duoman (Jijkoun and Hofmann Reference Jijkoun and Hofmann2009) and Pattern (De Smedt and Daelemans Reference De Smedt and Daelemans2012) sentiment lexicons. For English, we included the Liu and Hu opinion lexicon (Hu and Liu Reference Hu and Liu2004), the MPQA lexicon (Wilson, Wiebe, and Hoffmann Reference Wilson, Wiebe and Hoffmann2005), the General Inquirer Sentiment Lexicon (Stone et al. Reference Stone, Dunphy, Smith and Ogilvie1966), AFINN (Nielsen Reference Nielsen2011), and MSOL (Mohammad, Dunne, and Dorr Reference Mohammad, Dunne and Dorr2009). For both languages, we included the relative frequency of all 68 psychometric categories in the LIWC dictionary for English (Pennebaker et al. Reference Pennebaker, Francis and Booth2001) and Dutch (Zijlstra et al. Reference Zijlstra, Van Meerveld, Van Middendorp, Pennebaker and Geenen2004). We hypothesize that certain psychometric and sentiment categories are informative of different roles. For instance, we expect more positive opinion to be expressed by a bystander-defender, whereas a bully is more likely to express negative sentiment and LIWC affective categories such as anxiety and anger.

Topic model features: Topic models provide hidden semantic structures in collections of text and are used to uncover thematic information. The similarity score of a post to one of these models provides a score for how semantically similar the post is to the topics in that text collection. By making use of the Gensim topic modeling library (Rehurek and Sojka Reference Rehurek and Sojka2010), several Latent Dirichlet allocation (LDA) (Blei, Ng, and Jordan Reference Blei, Ng and Jordan2003) and latent semantic analysis (Deerwester et al. Reference Deerwester, Dumais, Furnas, Landauer and Harshman1990) topic models with varying granularity (k = 20, 50, 100, and 200) were trained on data corresponding to each fine-grained cyberbullying category (e.g., threats, defamations, insults, defenses). The topic models were trained on a background corpus (EN: 1,200,000 tokens, NL: 1,400,000 tokens) scraped with the BootCAT (Baroni and Bernardini Reference Baroni and Bernardini2004) web-corpus toolkit. Using BootCaT, we collected ASKfm posts from user profiles (different than the ones used for corpus compilation) using lists of manually determined seed words that are characteristic of the different cyberbullying categories based on the training data (e.g., “slut” for the insult category and “ignore” for the defense category). BootCat is crawler software that collects text from the web if it contains seed words. In this manner, we collect posts that are likely to be representative of the cyberbullying category corresponding to the characteristic seed words. The resulting background corpus was cleaned by removing non-English and non-Dutch text and stop words before creating topic models.

When applied to the training data, this resulted in 871,044 and 795,072 features for English and Dutch, respectively. To prevent features with greater numerical values to be attributed more weight in the model than features with smaller values, we used feature scaling, that is all feature values were scaled to the range [0, 1].

4.2 Classification algorithms

The tested multiclass classification algorithms include a linear-kernel SVM, a LR, a passive-aggressive (PA), and a stochastic gradient descent (SGD) classifier.Footnote ^c We chose these regularized linear classification models as this type of algorithms is typically fast enough for handling large datasets. The linear SVM makes use of the dual l2-loss implementation in LIBLINEAR (Fan et al. Reference Fan, Chang, Hsieh, Wang and Lin2008), which uses a one-vs-rest strategy for multiclass classification. We fixed the loss function to squared hinge. The LR classifier also uses a one-vs-rest scheme for multiclass and a coordinate descent algorithm. The PA classifier (Crammer et al. Reference Crammer, Dekel, Keshet, Shalev-Shwartz and Singer2006) is a type of online margin-based learning meant for large-scale data such as ours. We use the hinge loss as the loss function in PA, which is equivalent to the PA-I algorithm described in Crammer et al. (Reference Crammer, Dekel, Keshet, Shalev-Shwartz and Singer2006). Other regularized linear models using SGD learning were tested with a modified-huber loss function (Zhang Reference Zhang2004) which is the equivalent of a quadratically smoothed SVM with ${\gamma=2}$ and the perceptron loss function. We used ${10^6/n\,instances}$ iterations for SGD, as well as PA. Testing the effectiveness of online and batch algorithms such as SGD and PA are useful for large datasets, for dealing with memory restrictions or for use with streaming data (such as social media content).

Feature selection settings and classifier hyper-parameters for each classifier type were tuned in grid search in fivefold cross-validation. Table 3 gives an overview of the tested hyper-parameters for the different algorithms. The winning parametrization was chosen by means of macro-averaged ${F_1}$-score averaged over the folds.

Table 3.

Tested hyper-parameters for each step in our machine learning pipeline

4.3 Ensemble approaches

Apart from evaluating the above-described single-algorithm classifiers, we also investigated two ensemble learning techniques: a Voting classifier and a Cascading classifier. Ensemble learning has the potential to improve classification performance by combining different individual classifiers and additionally mitigates data imbalance (Galar et al. Reference Galar, Fernandez, Barrenechea, Bustince and Herrera2012).

We tuned an ensemble Voting classifier out of the three best-scoring classifiers. Hence, the final classification was determined by majority vote on the predicted label by each classifier.

We also tested a Cascading (also known as “multistage”) classifier approach in which the output of the first classifier for binary detection (i.e., bullying vs. not bullying) of cyberbullying instances is followed by a multiclass role classifier for the positive predictions (i.e., bullying posts). The first-stage classifier detects the presence of cyberbullying (i.e., bullying or not). The predicted cyberbullying instances serve as input for the second-stage classifier, which then predicts to which one out of three participant roles (i.e., harasser, victim, or bystander-defender) the post appears. In this manner, we test if it is worthwhile to first detect the presence of cyberbullying in a post and to proceed with the second classifier that assigns a participant role to the post. For both the Voting and the Cascading ensemble, the selection of best models is based on the average cross-validation and holdout ${F_1}$-scores of the tuned feature selection and resampling pipeline.

Our previous work on cyberbullying detection in ASKfm posts (Van Hee et al. Reference Van Hee, Jacobs, Emmery, Desmet, Lefever, Verhoeven, De Pauw, Daelemans and Hoste2018) revealed that linear-kernel SVMs work well for binary classification of cyberbullying posts. The highest-scoring classifier setup and hyper-parameters (i.e., out of single-algorithm and ensemble Voting approaches) were selected for the second-stage multiclass role classification in the Cascading ensemble: for Dutch, this was the LR classifier, while the Voting classifier performed best for English. The detection classifier and the second-stage classifier were tuned for the SVM hyper-parameters in Table 3. Hyper-parameter selection was optimized jointly for both stages. In this manner, we tested whether it is plausible to first automatically detect instances of cyberbullying and subsequently classify the participant roles of positive bullying instances.

4.4 Feature selection and resampling for imbalance

Due to the realistic nature of our dataset compared to previously used balanced studies, we encountered severe class imbalance or skew, which could negatively affect machine learning performance. In cyberbullying detection experiments on Twitter data, Al-Garadi et al. (Reference Al-Garadi, Varathan and Ravana2016) encounter the same data imbalance problem with a similar ratio of bullying to not bullying posts.

In line with Al-Garadi et al. (Reference Al-Garadi, Varathan and Ravana2016), we investigate feature selection and data resampling, two techniques that can be used to enhance classification performance in general, but in particular when faced with (severe) class imbalance (Japkowicz and Stephen Reference Japkowicz and Stephen2002).

For the multiclass role of classification experiments, we include random undersampling, which randomly removes instances of the majority classes to obtain a desired ratio with one or more minority classes. This technique is used to bias the classifier toward the minority class (i.e., bullying posts). After initial experimentation, we set the ratio to ${1/100}$, so that the majority classes were undersampled until the minority “bystander-defender” class became 1${\%}$ of the training set. Initial runs determined that a balanced ratio and a ratio of 10% and 5% consistently produced worse results, so a ratio of 1% was chosen. In cyberbullying detection experiments, Al-Garadi et al. (Reference Al-Garadi, Varathan and Ravana2016) have shown that synthetic super-sampling such as Synthetic Minority Oversampling Technique (SMOTE) effectively improve classifier performance. These algorithms create new observations of minority classes usually by means of some type of nearest neighbors or bootstrapping method. We tested SMOTE+Tomek and SMOTE+Edited Nearest Neighbours (Batista, Prati, and Monard Reference Batista, Prati and Monard2004), as well as ADASYN (He et al. Reference He, Bai, Garcia and Li2008) for synthetic super-sampling but found these algorithms to be too computationally expensive to work with our high-dimensional data.

For feature selection, we chose filter metrics that are used to characterize both the relevance and redundancy of variables (Guyon and Elisseeff Reference Guyon and Elisseeff2003). For feature filtering, we relied on ANOVA F-value and mutual information (MI) score as metrics. The MI implementation in Scikit-learn relies on nonparametric methods based on entropy estimation from k-nearest neighbors distances as described in Ross (Reference Ross2014) (${k= 3}$). Using grid searches over the different pipeline setups, we determined the feature filter scoring function and the percentile of retained features to construct the model.

Feature selection and undersampling were tuned and parameterized for each classification algorithm, except for the Voting and Cascading ensemble approaches where undersampling was not applied. We also investigated whether the sequential order of feature selection and data resampling had any effect on the classification performance. Each possible pipeline setup and parametrization were evaluated by means of fivefold cross-validation on our hold-in development set.

5.1 Pretrained models

For English, we ran text classification experiments using pretrained BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019), RoBERTa (Liu et al. Reference Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov2019), and XLNet (Yang et al. Reference Yang, Dai, Yang, Carbonell, Salakhutdinov and Le2019) models.

BERT is an attention-based auto-encoding sequence-to-sequence model using two unsupervised task objectives. The first task is word masking, where the masked language model (MLM) has to guess which word is masked in its position in the text. The second task is next sentence prediction (NSP) performed by predicting whether two sentences are subsequent in the corpus or randomly sampled from the corpus. The specific English BERT model used is the “bert-base-uncased” as provided with the original paperFootnote ^d pretrained on the 3.3B word Wikipedia + BookCorpus corpus. We prefer using a lowercase (“uncased”) model over a case-preserving (“cased”) model as our cyberbullying dataset contains social media text with many mistakes against capitalization convention. However, a lowercased pretrained model was only available for English BERT.

The RoBERTa model (Liu et al. Reference Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov2019) improved over BERT by dropping NSP and using only the MLM task on multiple sentences instead of single sentences. The authors argue that while NSP was intended to learn inter-sentence coherence, it actually learned topic similarity because of the random sampling of sentences in negative instances. The specific RoBERTa model used is the “roberta-base” cased model provided alongside the original paperFootnote ^e pretrained on the 160-GB Wikipedia + BooksCorpus + CommonCrawl-News + CommonCrawl-Stories + OpenTextWeb corpus.

XLNet is a permutation language model that combines strengths of auto-regressive and auto-encoding modeling approaches: permutation language models are trained to predict tokens given preceding context like a traditional unidirectional language model, but instead of predicting the tokens in sequential order, it predicts tokens in a random order sampling from both the left and right context. XLNet incorporates two key ideas from the TransformerXL architecture: relative positional embeddings and the recurrence mechanism. In combination with the permutation objective, these techniques effectively capture bidirectional context while avoiding the independence assumption and the discrepancy between pretraining and fine-tuning caused by the use of masked tokens in BERT. The specific XLNet model used is the “xlnet-base” cased model released alongside the original workFootnote ^f pretrained on a Wikipedia + BooksCorpus + Giga5 + ClueWeb 2012-B + Common Crawl totalling 339B SentencePiece tokens.

For Dutch, we tested BERTje (de Vries et al. Reference de Vries, van Cranenburgh, Bisazza, Caselli, van Noord and Nissim2019) and RobBERT (Delobelle, Winters, and Berendt Reference Delobelle, Winters and Berendt2020) which are the Dutch architectural equivalents of BERT and RobBERTa. The specific BERTje model is “bert-base-dutch-cased”Footnote ^g trained on a 2.4-B token Wikipedia + Books + SoNaR-500 + Web news + Wikipedia corpus. The specific RobBERT model is “robbert-base”Footnote ^h pretrained on the Dutch part of the OSCAR corpus (39 GB). The choice of these models was informed by their improved performance over another available monolingual Dutch Bert-NL modelFootnote ⁱ or the multilingual mBERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019) model. As subword token input BERT and BERTje uses WordPiece, RoBERTa, and RobBERT use byte-level Byte Pair Encoding (BPE), and XLNet uses SentencePiece (Kudo and Richardson Reference Kudo and Richardson2018).

6.1 Single-algorithm classifier results

As can be deduced from Table 4, for the English dataset, the LR classifier obtained the best results with ${F_1}$= ${55.47\%}$ in cross-validation and ${F_1}$= ${54.54\%}$ on the holdout test set. The runner-up is the SVM classifier which performs comparably but trades in recall. The PA algorithm obtains a remarkably high precision. All classifiers show a marked improvement over the majority and random baselines with ${F_1}$-scores of, respectively, ${24.43\%}$ and ${12.14\%}$.

Table 4.

Results (%) for role classification using a single linear classification algorithm approach. Boldface indicates highest score for the relevant metric

Similar results are observed for cyberbullying role classification on the Dutch dataset: the LR classifier obtained the best results in both cross-validation (${F_1}$= 54.03%) and on the holdout set (${F_1}$= 53.92%). The SVM classifier is again a close second and the other algorithms performed comparably. Like we observed for the English corpus, PA obtained the highest precision for a trade-off in recall. Here as well, the classifiers substantially outperformed the majority (${F_1}$= ${24.16\%}$) and random baseline (${F_1}$= ${12.55\%}$).

The algorithms do not seem to under- or over-fit as evidenced by the similar scores obtained through cross-validation and on the holdout test set. When recall is important, for instance in a semi-automated moderation context where the system assists human moderators by flagging posts for manual review, LR classification seems to be the desired approach. However, when precision is important, for instance in fully automated moderation applications, PA appears to be a good choice. SGD obtains the worst performance on both datasets.

6.2 Ensemble classifier results

In this section, we investigate the benefits of ensemble learning in two different types of ensemble classifiers: a Voting ensemble and a Cascading ensemble. The Voting ensemble classifier combines the three best-scoring classifiers by majority vote. Concretely, each individual algorithm provides a prediction and the majority vote is considered for the final prediction. For English, we combined LR, SVM, and SGD as the individual models. For Dutch, LR, SVM, and PA classifiers are considered. These were tuned by cross-validation with the same experimental settings and hyper-parameters as described earlier.

The Cascading ensemble consists of two stages where an SVM first predicts positive instances of cyberbullying. Second, we choose the best multiclass classifier to perform fine-grained role classification on these instances. The second-stage role classifier for English is the Voting classifier, whereas for Dutch it is the SVM. For both models, feature selection was applied. In this manner, we can examine whether it is worthwhile to detect if cyberbullying is present in the first step before classifying the participant role in the second step.

As shown in Table 5, for both languages, the Cascading approach outperforms the Voting classifier in terms of recall while the latter obtains better precision. The second-stage results shown in Table 6 differ from the results in Table 5 because the former shows the macro-averaged scores on the three bullying roles (i.e., “harasser,” “victim,” and “bystander-defender”) without the negative “not bullying” class, whereas the latter shows the macro-averaged scores on all four classes.

Table 5.

Results (%) for ensemble role classification, combining logistic regression, SVM, and SGD for English and logistic regression, SVM, and the passive aggressive classifier for Dutch. Boldface indicates the highest score for the relevant metric. All scores are macro-averaged

Table 6.

Results (%) for the initial cyberbullying detection stage and subsequent role classification stage on the holdout test set in the best-performing Cascading ensemble for the Dutch and English corpus. Macro-averaged scores are given for both stages; binary-averaged scores on the positive class are shown for the detection stage

Both ensemble classifiers outperform the best single classifier on the holdout test set (cf. Table 4). To conclude, ensemble methods obtain a higher recall than single-classifier approaches, but they provide marginal practical value taking into account their computational cost.

To examine the performance of the Cascade ensemble in closer detail, we present the scores obtained in the two stages of the classification in Table 6. The first detection stage obtains a macro-averaged ${F_1}$-score of 78.9% on the holdout test set for English and 79.39% for Dutch. Since this is a binary cyberbullying detection task, binary-averaged precision, recall, and ${F_1}$-score are also presented. These are the results for the positive bullying instances only. The detection scores (stage 1) slightly underperform scores obtained in previous research on cyberbullying detection in this dataset (Van Hee et al. Reference Van Hee, Jacobs, Emmery, Desmet, Lefever, Verhoeven, De Pauw, Daelemans and Hoste2018) using comparable linear classification algorithms (their binary-averaged ${F_1}$-score is 64% for English and 61% for Dutch). The second-stage role classifier obtains an ${F_1}$-score of 41.28% for English and 37.52% for Dutch. The results show the negative effect of both error percolation in the first stage and the difficulty of cyberbullying role classification.

However, in spite of the relatively poor performance of the second-stage classifier, final recall scores for role classification (in cross-validation) remain higher in Cascading ensembles when compared to the single-algorithm and Voting approaches.

6.3 Feature selection and resampling

To mitigate the effects of data imbalance, we experimented with feature selection and random undersampling techniques. We created pipelines with each enabled and in sequence with both possible orders, that is resampling followed by feature selection and inversely, feature selection followed by resampling. When looking at the individual classifiers, we found that random undersampling rarely improved results and due to the added computational complexity this step was left out when testing the ensemble methods. Table 7 presents the results obtained by the best-performing classification pipelines of the above experiments. The scores were more often improved by feature selection than by random undersampling. The ${F_1}$-score improvements are minimal and are in line with the findings of Al-Garadi et al. (Reference Al-Garadi, Varathan and Ravana2016) who report similar increases.

Table 7.

Results (%) of the best configurations for both languages. Boldface indicates the highest score for the relevant metric. All scores are macro-averaged

The best score for English is obtained when feature selection is included as preprocessing, followed by random undersampling and SGD (modified huber loss) with ${F_1}$= 56.42%. For Dutch, the Cascading ensemble with ANOVA F-score feature selection obtains the best score with ${F_1}$ = 54.37% on the holdout set. It should be noted however, that SGD obtains the worst performance in cross-validation. More robust results are obtained by the Cascading classifier, where the cross-validation and holdout scores are consistent and obtain the highest recall.

6.4 PLM results

Finally, we discuss the results of the PLM fine-tuning experiments. Table 7 presents all results obtained by the best-performing classification pipelines of the above experiments including transformer-based classifiers.

The default hyper-parameters and configurations were applied for all transformer models. The only hyper-parameter that was tuned in cross-validation is the number of training epochs. All Dutch and English models performed best with four epochs, except for the English XLNet model, which obtained the best results with eight epochs. The results of the cross-validation experiments in Table 7 show that, respectively, RoBERTa and RobBERT outperform the optimized single classifiers and the ensemble methods for English and Dutch, with ${F_1}$-scores of 60.25% and 54.92%, respectively.

We also observe a precision improvement over the Voting ensembles for both languages. After applying the best models to the holdout test set, however, we observe a remarkable decrease in the English RoBERTa model compared with the cross-validated scores and we see the model outperformed by BERT and XLNet.

Overall, the table shows a performance increase of about 5 points with RoBERTa compared to the Cascading model (i.e., ${F_1}$= 60.25% versus F${_1}$= 55.19%). For Dutch, the improvement is less outspoken, with RobBERT scoring about 1.6 points better than the best ensemble method (i.e., ${F_1}$= 54.92% versus ${F_1}$= 53.31%). The transformer-based models further show more balance between precision and recall than the single and ensemble classifiers for English, but they show larger differences for Dutch. This observation applies to the cross-validated results and the results on the holdout test set. In Section 7, we examine the results of the best models in closer detail and we discuss their performance on the different class labels.