publications

2024

1. 

   What Do Language Models Hear?

   Ngo, Jerry, and Kim, Yoon

   ACL 2024

   Abs PDF Website

   This work explores whether language models encode meaningfully grounded representations of sounds of objects. We learn a linear probe that retrieves the correct text representation of an object given a snippet of audio related to that object, where the sound representation is given by a pretrained audio model. This probe is trained via a contrastive loss that pushes the language representations and sound representations of an object to be close to one another. After training, the probe is tested on its ability to generalize to objects that were not seen during training. Across different language models and audio models, we find that the probe generalization is above chance in many cases, indicating that despite being trained only on raw text, language models encode grounded knowledge of sounds for some objects.

2023

1. 

   Is CLIP Fooled by Optical Illusions?

   Ngo, Jerry, Sankaranarayanan, Swami, and Isola, Phillip

   ICRL Tiny Papers 2023

   Abs PDF Poster Website

   Recent large machine learning models such as CLIP have shown impressive generalization performance for various perception tasks. In this work, we explore to what extent they model the human cognitive process. We focus our attention on how these models perceive optical illusions. We present a simple way to assess the effect by presenting illusions in the form of image and text prompts while observing the changes in models’ output under different illusory strengths. Our results show that CLIP can indeed be fooled by different types of illusions relating to lightness and geometry.