2026-04-07T11:47:15Zhttps://ebiltegia.mondragon.edu/oai/request

oai:ebiltegia.mondragon.edu:20.500.11984/139912026-04-01T08:13:11Zcom_20.500.11984_1143col_20.500.11984_1148

00925njm 22002777a 4500 dc Ugarte Querejeta, Miriam author Valle Entrena, Pablo author Parejo, Jose Antonio author Segura, Sergio author Arrieta, Aitor author 2025 Large Language Models (LLMs) have recently gained significant attention due to their ability to understand and generate sophisticated human-like content. However, ensuring their safety is paramount as they might provide harmful and unsafe responses. Existing LLM testing frameworks address various safety-related concerns (e.g., drugs, terrorism, animal abuse) but often face challenges due to unbalanced and obsolete datasets. In this paper, we present ASTRAL, a tool that automates the generation and execution of test cases (i.e., prompts) for testing the safety of LLMs. First, we introduce a novel black-box coverage criterion to generate balanced and diverse unsafe test inputs across a diverse set of safety categories as well as linguistic writing characteristics (i.e., different style and persuasive writing techniques). Second, we propose an LLM-based approach that leverages Retrieval Augmented Generation (RAG), few-shot prompting strategies and web browsing to generate up-to-date test inputs. Lastly, similar to current LLM test automation techniques, we leverage LLMs as test oracles to distinguish between safe and unsafe test outputs, allowing a fully automated testing approach. We conduct an extensive evaluation on well-known LLMs, revealing the following key findings: i) GPT3.5 outperforms other LLMs when acting as the test oracle, accurately detecting unsafe responses, and even surpassing more recent LLMs (e.g., GPT-4), as well as LLMs that are specifically tailored to detect unsafe LLM outputs (e.g., LlamaGuard); ii) the results confirm that our approach can uncover nearly twice as many unsafe LLM behaviors with the same number of test inputs compared to currently used static datasets; and iii) our black-box coverage criterion combined with web browsing can effectively guide the LLM on generating up-to-date unsafe test inputs, significantly increasing the number of unsafe LLM behaviors. 979-8-3315-0179-2 2833-9061 https://katalogoa.mondragon.edu/janium-bin/janium_login_opac.pl?find&ficha_no=200362 https://hdl.handle.net/20.500.11984/13991 Large Language Models ODS 9 Industria, innovación e infraestructura ODS 10 Reducción de las desigualdades ASTRAL: Automated Safety Testing of Large Language Models