Details
Abstract:
This will be a two-part talk.
In the first part, I will talk about experimental methods to functionally characterize the space of learnt solutions in LLMs and how it relates to generalization. Concretely, I will discuss two kinds of functions: low-rank functions and tree-shaped functions. I’ll present experiments to demonstrate the following results: 1) Layer selective low-rank reduction of learnt matrices in LLMs can improve performance on downstream language understanding tasks by over 30% points, and 2) Across training runs, learnt transformers that implement tree-shaped computations over their inputs exhibit superior compositional generalization, even when in-distribution accuracy is the same.
In the second part, I’ll present recent efforts on deciphering the structure of another black box language-like system: the naturally arising communication of sperm whales in the wild. I show how a language model trained on sperm whale calls can be used not just for future call prediction but to enable a deeper understanding of the structure and function of this unknown biological system. Specifically, I will demonstrate how sperm whale vocalizations are significantly more complex than previously believed—with both previously unknown combinatorial structure and context-dependent call modulation.
Bio: Pratyusha Sharma is a final-year PhD student at MIT, advised by Prof. Antonio Torralba and Prof. Jacob Andreas. Before this, she was an undergraduate student at IIT Delhi. She studies the interplay between language, sequential decision-making and intelligence in natural and AI systems. Her research is published in interdisciplinary journals like Nature Communications and in academic conferences across machine learning, natural language processing, robotics, and marine biology. Her research has also been featured in articles in the New York Times, National Geographic, BBC, Washington Post, etc. She was recently a speaker at TED AI and was selected as a Rising Star in EECS, Data Science, and GenAI.