I am Jing Gu, a Ph.D student at the University of California, Santa Cruz working with Prof. Xin (Eric) Wang. Previously I was a master student at the University of California, Davis working with Prof. Zhou Yu. I was in Google Research as an intern working in 2022 summer, and a research intern at Adobe in 2023.

News

[2023.10] Our workshop AVLR (Advances in Language and Vision Research) 2024 is accepted to be held at ACL 2024!
[2023.10] Our paper R2H has been accepted by EMNLP 2023!
[2023.09] PHOTOSWAP is accepted to NeurIPS 2023!
[2023.06] Our SlugJARVIS team wins the Third Place ($50,000) in the inaugural Alexa Prize SimBot Challenge! [Media Coverage]
[2022.11] One paper accepted to AACL Finding!
[2022.08] Our paper JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents is on arXiv!
[2022.05] Our SlugJARVIS team won the Alexa Prize SimBot Public Benchmark Challenge! [Media Coverage]
[2022.04] Invited Talk in Chinese about Vision-and-Language Navigation in AI Drive! [Video in Chinese]
[2022.03] One paper about Vision-and-Language Navigation Survey accepted to ACL 2022!
[2022.03] Our SlugJARVIS team received an Amazon Alexa Prize Award to work on Alexa Prize SimBot Challenge. [link]
[2021.09] Started Ph.D. journey at UCSC!
[2021.05] One short paper accepted to ACL 2021!
[2021.04] One paper accepted to EACL 2021!
[2020.11] One paper accepted to AAAI 2021!
[2020.09] One paper accepted to EMNLP Finding!
[2020.10] One paper accepted to EMNLP WNUT workshop!

Research

My research interests are mainly Computer Vision, Natural Langauge Processing, and Embodied AI.

Publication & Manuscript

  • Muffin or Chihuahua? Challenging Large Vision-Language Models with Multipanel VQA
    Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, Xin Eric Wang
    Under Review
    [Paper] [Website] [Code] [Data]

  • PHOTOSWAP-2: Personalized Subject Swapping via Appearance Adaptation
    Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang
    Under Review

  • PHOTOSWAP: Personalized Subject Swapping in Images
    Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang
    NeurIPS 2023
    [paper] [code] [website]

  • R2H: Building Multimodal Navigation Helpers that Respond to Help
    Yue Fan, Jing Gu, Kaizhi Zheng, Xin Eric Wang
    EMNLP 2023
    [paper]

  • SlugJARVIS: Multimodal Commonsense Knowledge-based Embodied AI for SimBot Challenge
    Jing Gu*, Kaizhi Zheng*, Kaiwen Zhou Yue Fan, Xuehai He Jialu Wang Zonglin Di, Xin Eric Wang
    Alexa Prize SimBot Challenge Proceedings
    [paper]

  • JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents
    Kaizhi Zheng*, Jing Gu*, Kaiwen Zhou*, Yue Fan*, Jialu Wang*, Zonglin Di, Xuehai He, Xin Eric Wang
    Preprint
    SoCal NLP 2022
    Winner Model of the Alexa Prize SimBot Public Benchmark Challenge [link]
    [paper]

  • Memformer: Memory-Augmented Transformer
    Qingyang Wu, Zhenzhong Lan, Jing Gu, Zhou Yu
    in Proc. of AACL 2022 Findings
    [paper]

  • Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
    Jing Gu*, Eliana Stefani, Qi Wu, Jesse Thomason, Xin Eric Wang
    in Proc. of ACL 2022
    [paper] [code]

  • A Tailored Pre-Training Model for Task-Oriented Dialog Generation
    Jing Gu*, Qingyang Wu*, Chongruo Wu, Weiyan Shi, Zhou Yu
    in Proc. of ACL 2021
    [paper] [code]

  • Perception Score, a Learned Metric for Open-ended Text Generation Evaluation
    Jing Gu, Qingyang Wu, Zhou Yu
    in Proc. of AAAI 2021
    [paper]

  • Data Annealing for Informal Language Understanding Tasks
    Jing Gu, Zhou Yu
    in Proc. of EMNLP 2020 Findings
    [paper]

  • Flow-Aware Structual Model for Conversational Question Generation
    Jing Gu, Mostafa Mirshekari, Zhou Yu, Aaron Sisto
    in Proc. of EACL 2021 [paper] [code]

  • ConQuest: Contextual Question Paraphrasing through Answer-Aware Synthetic Question Generation
    Mostafa Mirshekari, Jing Gu, Aaron Sisto
    EMNLP 2021 WNUT Workshop [paper]

Service

Program Committe Member (Reviewer)

  • ACL 2021, Sigdial 2021, EMNLP 2021, NeurIPS 2021, ICLR 2022, ACL Rolling Review 2021&2022, NeurIPS 2022, EMNLP 2022, ACL 2023, ICML 2023, IEEE RA-L 23, EMNLP 2023, NeurIPS 2023

Honors & Awards

Misc

  • Teaching Assitant for Machine Learning, 2021
  • Teaching Assitant for Natural Language Processing, 2019
  • Teaching Assitant for Machine Dependent Programming, 2020
  • Research Scientist in Searchable AI Corp., 2020

One last word

Keep Hungry!