In the ever-evolving landscape of cybersecurity, phishing remains a formidable threat, often bypassing technological defences by exploiting human behaviour. Traditional warning dialogues, while common, often fall short due to their lack of explanatory clarity and static content. A recent study conducted by researchers Federico Maria Cau, Giuseppe Desolda, Francesco Greco, Lucio Davide Spano, and Luca Viganò explores the potential of Large Language Models (LLMs) to generate clear, concise, and scalable explanations for phishing warnings. Their findings suggest that LLMs could revolutionise how we protect users from phishing attacks.
The study, titled “Can Large Language Models Automate Phishing Warning Explanations? A Controlled Experiment on Effectiveness and User Perception,” involved a large-scale between-subjects user study with 750 participants. The researchers compared the effectiveness of warning dialogues supplemented with manually generated explanations against those generated by two LLMs: Claude 3.5 Sonnet and Llama 3.3 70B. They investigated two explanatory styles—feature-based and counterfactual—to assess their impact on behavioural metrics, such as click-through rates, and perceptual outcomes, including trust, risk, and clarity.
The results of the study provide compelling evidence that LLM-generated explanations can achieve a level of protection statistically comparable to expert-crafted messages. This finding is significant as it demonstrates the potential for automating a high-cost task, making phishing warnings more scalable and adaptable. Claude 3.5 Sonnet, in particular, showed a trend towards reducing click-through rates compared to manual baselines, indicating its effectiveness in deterring users from phishing attempts. However, Llama 3.3, despite being perceived as clearer, did not yield the same behavioural benefits.
The study also revealed that feature-based explanations were more effective for genuine phishing attempts, while counterfactual explanations helped in diminishing false-positive rates. This nuanced understanding of explanatory styles can guide the development of more effective phishing warning systems. Additionally, the researchers found that variables such as workload, gender, and prior familiarity with warning dialogues significantly moderated the effectiveness of warnings. This highlights the importance of tailoring phishing warnings to different user demographics and contexts.
The implications of this research extend beyond immediate applications in cybersecurity. By leveraging LLMs, organisations can create more dynamic and personalised warning systems that adapt to the evolving tactics of cybercriminals. This not only enhances user protection but also reduces the burden on cybersecurity experts, allowing them to focus on more strategic tasks. The study underscores the potential of LLMs to automate high-cost tasks while maintaining a high level of effectiveness, aligning with human-centred values.
As cyber threats continue to grow in sophistication, the need for innovative and scalable solutions becomes ever more critical. The findings of this study offer a promising avenue for enhancing phishing warning systems, ultimately contributing to a safer digital environment. By integrating LLMs into cybersecurity frameworks, we can build more resilient defences that protect users without compromising on clarity or effectiveness. This research marks a significant step forward in the ongoing battle against phishing, demonstrating the transformative potential of artificial intelligence in safeguarding our digital world. Read the original research paper here.