Title:  Evaluating the Social Impact of Generative AI Systems in Systems and Society

Irene Solaiman, Jennifer Mickel, and Usman Gohar, Hugging Face
View via Livestream: https://youtube.com/live/pBShTHNDO-w

Abstract: Generative AI systems across modalities, ranging from text, image, audio, video, and multimodal, have broad social impacts, but there exists no official standard for means of evaluating those impacts and which impacts should be evaluated. We move toward a standard approach in evaluating a generative AI system for any modality, in two overarching categories: what is able to be evaluated in a base system that has no predetermined application and what is able to be evaluated in society. We describe specific social impact categories and how to approach and conduct evaluations in the base technical system, then in people and society. We are currently crafting an evaluation repository for the AI research community and examining methods for evaluating evaluations.

Speaker Bios:

Usman Gohar is a Ph.D. student and an F. Wendell Miller Scholar in the Department of Computer Science at Iowa State University. His research broadly lies at the intersection of Software Engineering and Machine Learning, with a particular focus on AI Safety, Algorithmic Fairness, Social Computing, and AI for Social Good. His research includes developing techniques to mitigate algorithmic unfairness, conducting fairness evaluations and developing a fair and safety-aware ecosystem for drones.

Jennifer Mickel studied computer science and mathematics as a Turing Scholar and Polymathic Scholar at the University of Texas at Austin, where she was a Dean’s Honored Graduate. Her research lies in AI and algorithmic fairness, machine learning, and NLP. Her research focuses on understanding societal biases within LLMs, how they present in downstream tasks, and their effect on users, as well as developing algorithms and frameworks to mitigate and address these biases within existing systems and in downstream tasks.

Irene Solaiman is an AI safety and policy expert. She is Head of Global Policy at Hugging Face, where she is conducting social impact research and leading public policy. Irene serves on the Partnership on AI's Policy Steering Committee and the Center for Democracy and Technology's AI Governance Lab Advisory Committee. Irene advises responsible AI initiatives at OECD and IEEE. Her research includes AI value alignment, responsible releases, and combating misuse and malicious use. Irene was recently named and was named MIT Tech Review's 35 Innovators Under 35 2023 for her research.  Irene formerly initiated and led bias and social impact research at OpenAI, where she also led public policy. Her research on adapting GPT-3 behavior received a spotlight at NeurIPS 2021. She was recently also Tech Ethics and Policy Mentor at Stanford University and an International Strategy Forum Fellow at Schmidt Futures. She formerly built AI policy at Zillow Group and advised policymakers on responsible autonomous decision-making and privacy as a fellow at Harvard’s Berkman Klein Center.