Dr Catherine O’Brien, Dr Mark Warner, Dr Amid Ayobi (UCLIC), and former MSc student Izzy Ferreira-Alturas published a paper at NordiCHI that reports on co-design sessions with Instagram users, to design an LLM-based user reporting tool for reporting ambiguous and targeted harassment.
Ambiguous harassment poses a challenge for platforms as it exploits plausible deniability and platform opacity, leaving users uncertain about what qualifies as reportable. We present findings from a co-design study with UK Instagram users investigating how Large Language Models (LLMs) might support user reporting of ambiguously harassing content, and what they need from an LLM-facilitated reporting tool in this context.
Our findings point to opportunities for LLMs to prompt reflection, helping to shift reporting away from current categorical approaches. We highlight how LLM integration could help embed educational and platform policy resources within reporting interaction flows, supporting users in interpreting and describing harassment relative to platform policies, and potentially offer personalised aftercare and wellbeing support to safeguard against one-off and ongoing harassment. Balanced with considered ethical implications, we use our findings to develop design implications that promote LLM-based interactions that prioritise user support over content classification, help validate lived experience, counter platform-level gaslighting, and address identity-based disparities while preserving user autonomy and control.