View Proposal

---

Proposer

John See

Title

Talk to the Museum: Multimodal Retrieval Augmented Generation for Museums

Goal

A chat application that allows museum visitors to query for more information about things in a museum or heritage site using text and pictures

Description

Exploring new interactions between visitors and the museum may be vital to increase the appeal of museums (and heritage sites) for the new generation. Traditionally, museums tend to depend a lot on manually disseminated information such as site experts and tour guides, as well as using specific types of technologies like sensor-based audio/visual guides and on-site augmented reality to make things interesting. Recent advances in AI-based chatbots now present new possibilities to museums. Wouldn't it be fascinating if visitors could ask questions to the chatbot about the historical facts of a specific artifact in the museum? Can visitors ask the chatbot about a certain picture of a specific pattern on the exhibit? This project should explore the use of retrieval augmented generation (RAG), which combines the strengths of traditional information retrieval systems with the capabilities of generative large language models (LLMs), build a demonstrable prototype, and have it evaluated by early users.

Resources

Free cloud compute (Colab) should suffice. Application can be hosted locally or publicly (HuggingFace, etc.)

Background

Reasonable level of competency in programming, especially Python; Some familiarity with LLMs and API integration in apps would be an added advantage;

Url

Difficulty Level

Moderate

Ethical Approval

InterfaceOnly

Number Of Students

1

Supervisor

John See

Keywords

multimodal llms, retrieval augmented generation, generative ai, multimedia for heritage

Degrees

Bachelor of Science in Computing Science