View Proposal

---

Proposer

Oliver Lemon

Title

Conversational speech and multimodal Interfaces for Role-playing games using Generative AI

Goal

Build and evaluate a speech or multimodal (language + vision) interface for an RPG, using LLMs or VLMs

Description

There are interesting opportunities in adding speech and dialogue capabilities into video games -- for example making conversational game characters that you can really talk to, or a companion character that can assist and support the player. Such characters need to be believable, safe, and responsive to the game world. Evaluation frameworks (benchmarks) for such characters are also an open area of investigation. A related area is in using generative models to create interactive narratives/stories for players, which are coherent and immersive. New LLMs (large language models)and VLMs such as GPT 4o and Gemini are now being used to create games and conversational game characters, see Mantella and CHIM : https://www.nexusmods.com/skyrimspecialedition/mods/98631 , https://www.nexusmods.com/skyrimspecialedition/mods/126330 This project could explore such systems and LLMs and integrate them a into a game engine (e.g. Skyrim, Minecraft, etc) . For example to drive conversations with NPCs, or develop a visually-aware conversational game companion, search lore etc. See e.g. https://www.youtube.com/watch?v=OiPZpqoLs4E and https://www.youtube.com/watch?v=tVd3QYc0fU8 You will evaluate aspects such as the usability, safety, and immersion value of this system versus the baseline game. This project can be done in collaboration with an industrial partner, for example SpeechGraphics, and is also suitable for DataLab students. See https://arxiv.org/html/2402.18659v1 for a recent survey of this area. You should take the course F20/21CA Conversational Agents.

Resources

A moddable RPG, e.g. Skyrim; Witcher; Fallout, Minecraft, etc . Mantella https://www.nexusmods.com/skyrimspecialedition/mods/98631 CHIM https://www.nexusmods.com/skyrimspecialedition/mods/126330

Background

AI, speech, computer vision, and/or games design and development

Url

External Link

Difficulty Level

Moderate

Ethical Approval

InterfaceOnly

Number Of Students

3

Supervisor

Oliver Lemon

Keywords

ai, games, generative ai, llm, vlm, speech, multimodal, evaluation

Degrees

Bachelor of Science in Computer Science
Master of Engineering in Software Engineering
Master of Design in Games Design and Development
Master of Science in Artificial Intelligence
Master of Science in Artificial Intelligence with SMI
Master of Science in Computing (2 Years)
Master of Science in Data Science
Master of Science in Human Robot Interaction
Master of Science in Robotics
Master of Science in Software Engineering
Bachelor of Science in Computing Science
Bachelor of Engineering in Robotics
Master of Science in Robotics with Industrial Application
Postgraduate Diploma in Artificial Intelligence