View Proposal

---

Proposer

Simona Frenda

Title

Evaluation of inclusivity level of models

Goal

Design new evaluation metrics and/or frameworks to measure how aware models are of diverse perspectives on the same phenomenon (e.g., irony, hate speech, emotions, misinformation) in linguistic data.

Description

Looking deeply at the data annotated especially for recognizing pragmatic phenomena, such as irony and hate speech detection, human label variation unveils different points of view [1]. In order to take into account all the perspectives coming from data annotation, scholars have designed perspectives-aware models trained on multiple and disaggregated corpora [2]. Evaluation of these models is still an open challenge [3]: they should be evaluated against test sets that include and reflect the diverse perspectives mined in the data and not on a single "ground truth" (generally obtained by majority voting) [4]. Therefore, in this project, we will explore existing metrics used to evaluate the comprehension of models in specific cultural contexts with multilingual textual dataset, identify limitation of these metrics, open new challenges and plan new strategies to evaluate its level of inclusivity.

Resources

[1] Plank B. (2022) The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. [2] Abercrombie, G., Basile, V., Bernadi, D., Dudy, S., Frenda, S., Havens, L., & Tonelli, S. (2024). Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives)@ LREC-COLING 2024. [3] Frenda, S., Abercrombie, G., Basile, V., Pedrani, A., Panizzon, R., Cignarella, A.T., Marco, C. and Bernardi, D., (2024). Perspectivist approaches to natural language processing: a survey. Language Resources and Evaluation. [4] Mokhberian, N., Marmarelis, M. G., Hopp, F. R., Basile, V., Morstatter, F., & Lerman, K. (2023). Capturing perspectives of crowdsourced annotators in subjective learning tasks. arXiv preprint arXiv:2311.09743.

Background

https://pdai.info/ https://nlperspectives.di.unito.it/

Url

Difficulty Level

Moderate

Ethical Approval

None

Number Of Students

0

Supervisor

Simona Frenda

Keywords

evaluation, perspectivist models, multiple annotated corpora, multilingual dataset

Degrees

Bachelor of Science in Computer Science
Master of Science in Artificial Intelligence
Master of Science in Artificial Intelligence with SMI
Master of Science in Data Science
Bachelor of Science in Computing Science
BSc Data Sciences