projects

2021

1. 

   The Effect Of Data Augmentation on Deep Representations

   Phuc Ngo, Saachi Jain, and Mądry, Aleksander

   2021

   Abs Poster Slides

   Data augmentation is a simple and common technique that increases the model’s robustness to class-preserving transformations. However, our understanding of how data augmentation affects deep representations is limited. In this work, we attempt to study this further and hypothesize two mechanisms that could happen. The earlier layers of the model could map augmented inputs to similar representations to the standard inputs counterpart. Or, the model could use an entirely different set of prediction rules to classify augmented samples. To test the hypothesis, we trained standard and augmented models to analyze the similarity between their predictions and representations. Our results suggest data augmentation has a range of behavior on deep representations. Depending on the severity of the augmentation, models can vary between learning invariance or learning entirely separate augmented subpopulations.

2. 

   StyleWav: Guiding Image Synthesis Using Audio

   Phuc Ngo, Phillip Isola

   2021

   Abs PDF Slides

   Recent works in multimodality systems have enabled image gener- ations with instruction from another domain. Specifically, Contrastive Language-Image Pre-training (CLIP) allows multiple downstream tasks such as image synthesis, 3D object generation, and style transfer using text. However, the audio-image generation has not been much explored. In this work, we introduce Style-Wav, an algorithm that is adapted from StyleCLIP. Instead of using text guidance as in StyleCLIP, StyleWav uses audio representation from Wav2CLIP to guide the generated image using StyleGAN. We use synthesized images as a concrete benchmark for the transferred knowledge from the distillation process from CLIP to the audio domain. For further application, the model’s code is public at:https://github.com/Jerry2001/StyleWav.

3. 

   An Application of NEAT and HyperNEAT in Solving A Sliding Tile Puzzle

   Phuc Ngo, Mehmet Dik

   2021

   Abs PDF Slides

   Neuroevolution is a set of algorithms that use evolutionary algorithms to optimize neural networks without much domain knowledge. We analyze Neuroevolution of Augmented Topologies (NEAT) and its extension HyperNeat in this paper. NEAT evolves both the topology and weight values of a network along with novel ideas of applying speciation, tracking genes, and evolving from simple structures. HyperNEAT uses similar techniques to evolve networks but instead of using direct graph encoding as in NEAT, it uses indirect graph encoding. We use a stochastic single-player game, 2048, as the benchmark problem to compare two algorithms’ performance. Even though the game is simple, it has the random factor that may pose a challenge in finding a strategy to achieve a high score. The paper analyzes the strategy and the performance of NEAT and HyperNEAT in 2048 with different parameter settings. Furthermore, code and future work are specified at the end of the paper.