Welcome to the PLI Blog! Our blog post will cover topics such as new AI research, cross-disciplinary applications, societal implications, and more!

About PLI BLog

BitDelta: Your Fine-Tune May Only Be Worth One Bit
Feb. 20, 2024

James Liu1*
Guangxuan Xiao1
Kai Li2
Jason D. Lee2
Song Han1
Tri Dao2,3
Tianle Cai2,3*

*indicates equal contribution
1MIT, 2Princeton University 3Together AI

ALiBi FlashAttention - Speeding up ALiBi by 3-5x with a hardware-efficient implementation
Jan. 25, 2024

ALiBi is a simple and widely used method to improve Transformer quality and length extrapolation. With a hardware-efficient implementation, we speed up ALiBi by 3-5x, unlocking new use cases of ALiBi for large-scale training.

Black-Box Detection of Pretraining Data
Jan. 4, 2024
Black-Box Detection of Pretraining Data Background

2023 has witnessed the proliferation of strong large language models (LLMs) like GPT-4, LLaMA-2, Mistral, etc. These models are typically trained using corpora consisting of trillions of tokens scraped from internet web pages. As the scale of the data used to pretrain LLMs has risen to…

Princeton Language and Intelligence (PLI) at NeurIPS 2023
Dec. 11, 2023

NeurIPS, the top AI and ML conference, is being held in New Orleans from December 10 to December 16. Princeton students, postdocs, and faculty will be there presenting a wide array of work. This post summarizes Princeton research being presented at the conference, highlighting the breadth of machine learning…

Princeton Language and Intelligence (PLI) at EMNLP 2023
Dec. 7, 2023

The 2023 Conference on Empirical Methods in Natural Language Processing is being held in Singapore from the 6th of December to the 10th of December 2023.

We are excited to announce that 15 main conference/findings papers authored by Princeton researchers will be presented at…

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Dec. 6, 2023

This post is based on the following work:
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez*1, John Yang*1, Alexander Wettig1, Shunyu Yao1, Kexin Pei2, Ofir Press1, Karthik Narasimhan1
1Princeton Language…

Are Language Models Mere Stochastic Parrots? The SkillMix Test Says NO.
Nov. 28, 2023

Our new SKILLMIX evaluation tests a simple form of compositional generalization—the ability to mix and match linguistic skills to achieve narrative goals. This is arguably an essential component of human language.  The evaluation starts with a list of $N$ skills that every LLM ought to know (currently our list includes  linguistic, reasoning, or rhetorical skills that have a name and a Wikipedia entry) and a set of $T$ topics that have low, but non-negligible, probability, such as “sewing.” It randomly selects a topic and a random subset of $k$ skills, and asks the model to produce a piece of text on that topic with at most $k-1$ sentences and exhibiting all $k$ skills.

Welcome to the PLI Blog!
Nov. 28, 2023

We are excited to announce the launch of the PLI Blog. PLI is committed to keeping AI expertise and know-how in the public sphere.