Towards Multimodal, Multicultural, and Multilingual Understanding

In this talk, Graham Neubig will discuss a new frontier in AI models, vision-language models that understand the world's cultures. The talk will be in two parts. First, Neubig will discuss training of multilingual multimodal multicultural models that understand images and text, and have increased ability to answer culture-specific questions about multimodal data. Second, he will discuss work on "image transcreation", where models have been developed that can transform images to make them more relevant to a particular culture. This work has applications in a number of areas, such as cultural localization of educational materials (to accompany translated text). While these methods cover many languages, African and not, the talk will focus on examples specifically from the African context, and challenges we currently face therein.

Graham Neubig is an associate professor at the Language Technologies Institute of Carnegie Mellon University. His research focuses on natural language processing, with a particular interest in fundamentals, applications, and understanding of large language models for tasks such as question answering, code generation, and multilingual applications. His final goal is that every person in the world should be able to communicate with each other, and with computers in their own language. He also contributes to making NLP research more accessible through open publishing of research papers, advanced NLP course materials and video lectures, and open-source software, all of which are available on his website.