Yue Yang
pronounced as yoo-eh

220 South 33rd Street, Towne 299
Philadelphia, PA 19104, USA
Email: yueyang1 [at] seas.upenn.edu

Google Scholar Icon Semantic Scholar Icon GitHub Icon Twitter Icon LinkedIn Icon Youtube Icon Pixiv Icon CV

About me

Hi! My name is Yue Yang (杨樾). I am a final-year Ph.D. student in Computer and Information Science at the University of Pennsylvania, affiliated with Penn NLP. I am grateful to be advised by Prof. Chris Callison-Burch and Prof. Mark Yatskar. I am interested in the intersection area of Natural Language Processing (NLP) and Computer Vision (CV).

My current research focuses on applying the knowledge priors of large language models (LLMs) to various domains (images, videos, healthcare, Embodied AI, etc) to improve different aspects of AI systems, including:

Interpretability. LLMs aid in constructing human-readable intermediate representations, such as concept bottlenecks, enabling the design of inherently interpretable models, thereby mitigating the black-box nature of deep learning.

Robustness. By utilizing sparse natural language representations as input, models are less prone to overfitting on the spurious cues of in-domain training data, enhancing their robustness and out-of-domain generalization.

Data Efficiency. Leveraging the world knowledge and coding ability of text-only LLMs to create synthetic data to improve embodied agents and multimodal language models.

I am looking for full-time positions starting in Summer 2025. Please reach out if you are interested in working with me!


Selected Works
Matt Deitke*, Christopher Clark*, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, et al. (51 authors in total)
Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael S. Yao, Chris Callison-Burch, James C. Gee, Mark Yatskar
Conference on Neural Information Processing Systems (NeurIPS), 2024 (Spotlight)
TL;DR: We introduce Knowledge Bottlenecks (KnoBo) that incorporate priors from medical documents, such as PubMed, through inherently interpretable models. KnoBo is robust to domain shifts in medical images, e.g., data sampled from different hospitals or data confounded by demographic variables such as sex, race, etc. Overall, our work demonstrates that a key missing ingredient for robustness to distribution shift in medical imaging models is a prior rooted in knowledge.
a 1b1b apartment of a researcher who has a cat
Yue Yang*, Fan-Yun Sun*, Luca Weihs*, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison-Burch, Mark Yatskar, Aniruddha Kembhavi, Christopher Clark
Conference on Computer Vision and Pattern Recognition (CVPR), 2024
TL;DR: Holodeck is an automated system for generating diverse 3D environments in Embodied AI, using a large language model (GPT-4) and a vast collection of 3D assets from Objaverse. It can create complex scenes based on user prompts, adjusting for styles and specific details, like "a 1b1b apartment of a researcher who has a cat".
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, Mark Yatskar
Conference on Computer Vision and Pattern Recognition (CVPR), 2023
TL;DR: Concept Bottleneck Models are interpretable models that factor in human-readable concepts to explain model decisions. However, CBMs often under-perform their black box counterparts and require manual specification of concepts. Our method, LaBo, leverages large language models (GPT-3) to automatically construct bottlenecks for any image classification tasks.

Education
University of Pennsylvania, Philadelphia, PA, USA
  • Ph.D. in Computer and Information Science (2020 - present)
  • M.S. in Robotics (2018 - 2020)
  • Zhejiang University, Hangzhou, China
  • B.E. in Mechanical Engineering (2014 - 2018)

  • Experiences
    Allen Institute for AI, Seattle, WA, USA
    Research Intern (May. 2023 to Sept. 2023, May. 2024 to Sept. 2024)
    Outstanding Intern of the Year Award (2023)
    Tencent AI Lab, Seattle, WA, USA
    Research Scientist Intern (May. 2022 to Sept. 2022)

    Teaching

    Head Teaching Assistant, CIS-521 Artificial Intelligence, University of Pennsylvania
    Fall2019; Fall 2020; Summer 2021; Fall 2021; Spring 2022

    Teaching Assistant, CIS-530 Computational Linguistics, University of Pennsylvania
    Spring 2021

    Academic Service

    Reviewer:
    Computer Vision: CVPR, ECCV, SIGGRAPH Asia.
    Natural Language Processing: ACL, EMNLP, NAACL, EACL, COLM.
    Machine Learning: NeurIPS, TMLR, ICLR.

    Talks
    WPE-II Presentation, University of Pennsylvania, Philadelphia, PA, USA
    Language Guided Concept Bottlenecks for Interpretable and Robust Image Classification, April 29, 2024. slides
    CLUNCH, University of Pennsylvania, Philadelphia, PA, USA
    Investigate Procedural Events in a Multimodal Fashion, November 22, 2021. slides

    Website source from Jon Barron.