AI Seminar Series - Pierre Peigné

We introduce the PRISM Eval Behavior Elicitation Tool (BET), an AI system that conduct LLMs automated red-teaming through dynamic adversarial optimization. BET outperforms other dynamic adversarial optimization systems in terms of Attack Success Rate (ASR) and provides a new finer grained robustness metric based on the average number or attempts required to jailbreak a system. In addition to this, BET also allows to map and study the vulnerability landscape of any LLM system leading to new findings.

Finally, we present a leaderboard of LLM robustness using BET, where 39 SotA LLMs (including the Falcon 3 10b!) were evaluated against 5 categories of harms.