Jason D. Lee2
*indicates equal contribution
1MIT, 2Princeton University 3Together AI
ALiBi is a simple and widely used method to improve Transformer quality and length extrapolation. With a hardware-efficient implementation, we speed up ALiBi by 3-5x, unlocking new use cases of ALiBi for large-scale training.
2023 has witnessed the proliferation of strong large language models (LLMs) like GPT-4, LLaMA-2, Mistral, etc. These models are typically trained using corpora consisting of trillions of tokens scraped from internet web pages. As the scale of the data used to pretrain LLMs has risen to…
NeurIPS, the top AI and ML conference, is being held in New Orleans from December 10 to December 16. Princeton students, postdocs, and faculty will be there presenting a wide array of work. This post summarizes Princeton research being presented at the conference, highlighting the breadth of machine learning…
The 2023 Conference on Empirical Methods in Natural Language Processing is being held in Singapore from the 6th of December to the 10th of December 2023.
We are excited to announce that 15 main conference/findings papers authored by Princeton researchers will be presented at…
This post is based on the following work:
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez*1, John Yang*1, Alexander Wettig1, Shunyu Yao1, Kexin Pei2, Ofir Press1, Karthik Narasimhan1
Our new SKILLMIX evaluation tests a simple form of compositional generalization—the ability to mix and match linguistic skills to achieve narrative goals. This is arguably an essential component of human language. The evaluation starts with a list of $N$ skills that every LLM ought to know (currently our list includes linguistic, reasoning, or rhetorical skills that have a name and a Wikipedia entry) and a set of $T$ topics that have low, but non-negligible, probability, such as “sewing.” It randomly selects a topic and a random subset of $k$ skills, and asks the model to produce a piece of text on that topic with at most $k-1$ sentences and exhibiting all $k$ skills.
We are excited to announce the launch of the PLI Blog. PLI is committed to keeping AI expertise and know-how in the public sphere.