NPC-Bench: A Benchmark Dataset for Immersion and Safety of Generative AI for Non-player Characters

We construct a robust and diverse benchmark test set to comprehensively evaluate language models in the context of natural language interactions with non-player characters (NPCs) for role-playing games. Such models need to be able to stay in character, stay consistent with the game world, and answer appropriately when confronted with real-world events and entities which the character should not be aware of (e.g., cars or Taylor Swift for a medieval setting). In addition, such characters need to be safe, avoiding harmful or toxic content. Results indicate that our novel prompting strategy led to improved performance on this new benchmark, for example showing a 6% improvement for Gemini Flash 2.0, achieving an accuracy of 94%. In addition, a study with real users showed improvements across a variety of gameplay dimensions.