Jerry Ngo

ngop at mit dot edu

I am a research associate at McGovern Institute and MIT BCS, supervised by Jim DiCarlo. During my undergraduate, my research journey took root at MIT CSAIL, where I still involve as a frequent collaborator. At CSAIL, I was fortunate to be supervised by Phillip Isola, Yoon Kim, and Aleksander Mądry.

With a broad research goal of understanding AI, I ground my works at the intersection of computer vision and neuro/cognitive science. Currently, my projects are focused on quantifying behavioral similarities and disparities between vision models and human behavior (1, 2). I am also interested in other topics like multimodality/representation learning, and generative modeling.

news

May 16, 2024	One paper got accepted at ACL 2024!!.
Apr 10, 2023	First paper got accepted for publication at ICLR 2023 Tiny Papers track.
Jan 6, 2023	Join the MIT BCS as a research technician.
Jun 27, 2022	Join the MIT Summer Research Program ‘22.
Jan 3, 2022	Join MIT CSAIL as a visiting student.

selected publications

“A journey of a thousand papers begins with a tiny paper”

What Do Language Models Hear?

Ngo, Jerry, and Kim, Yoon

ACL 2024

Abs PDF Website

This work explores whether language models encode meaningfully grounded representations of sounds of objects. We learn a linear probe that retrieves the correct text representation of an object given a snippet of audio related to that object, where the sound representation is given by a pretrained audio model. This probe is trained via a contrastive loss that pushes the language representations and sound representations of an object to be close to one another. After training, the probe is tested on its ability to generalize to objects that were not seen during training. Across different language models and audio models, we find that the probe generalization is above chance in many cases, indicating that despite being trained only on raw text, language models encode grounded knowledge of sounds for some objects.
Is CLIP Fooled by Optical Illusions?

Ngo, Jerry, Sankaranarayanan, Swami, and Isola, Phillip

ICRL Tiny Papers 2023

Abs PDF Poster Website

Recent large machine learning models such as CLIP have shown impressive generalization performance for various perception tasks. In this work, we explore to what extent they model the human cognitive process. We focus our attention on how these models perceive optical illusions. We present a simple way to assess the effect by presenting illusions in the form of image and text prompts while observing the changes in models’ output under different illusory strengths. Our results show that CLIP can indeed be fooled by different types of illusions relating to lightness and geometry.