I am Jing Gu, a Ph.D student at the University of California, Santa Cruz working with Prof. Xin (Eric) Wang. Previously I was a master student at the University of California, Davis working with Prof. Zhou Yu. Previously I was a research intern in Google Research, Adobe, Nvidia Research.

News

[2024.07] SwapAnything is accepted to ECCV with all positive score! [Bilibili Video]
[2024.06] Invited Talk at ByteDance AI Lab on Advancing Visual and Video Editing for Creative Empowerment.
[2023.12] Invited Talk at Prof. Huaizu Jiang’s Lab on Personalized Visual Editing.
[2023.10] Our workshop AVLR (Advances in Language and Vision Research) 2024 is accepted to be held at ACL 2024!
[2023.10] Our paper R2H has been accepted by EMNLP 2023!
[2023.09] PHOTOSWAP is accepted to NeurIPS 2023!
[2023.06] Our SlugJARVIS team wins the Third Place ($50,000) in the inaugural Alexa Prize SimBot Challenge! [Media Coverage]
[2022.11] One paper accepted to AACL Finding!
[2022.08] Our paper JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents is on arXiv!
[2022.05] Our SlugJARVIS team won the Alexa Prize SimBot Public Benchmark Challenge! [Media Coverage]
[2022.04] Invited Talk in Chinese about Vision-and-Language Navigation in AI Drive! [Video in Chinese]
[2022.03] One paper about Vision-and-Language Navigation Survey accepted to ACL 2022!
[2022.03] Our SlugJARVIS team received an Amazon Alexa Prize Award to work on Alexa Prize SimBot Challenge. [link]
[2021.09] Started Ph.D. journey at UCSC!
[2021.05] One short paper accepted to ACL 2021!
[2021.04] One paper accepted to EACL 2021!
[2020.11] One paper accepted to AAAI 2021!
[2020.09] One paper accepted to EMNLP Finding!
[2020.10] One paper accepted to EMNLP WNUT workshop!

Research

My research interests are mainly Computer Vision, Natural Langauge Processing, and Embodied AI.

Publication & Manuscript

• VIA: A Spatiotemporal Video Adaptation Framework for Global and Local Video Editing
  Jing Gu, Yuwei Fang, Ivan Skorokhodov, Peter Wonka, Xinya Du, Sergey Tulyakov, Xin Eric Wang
  Arxiv
  [Paper] [Website] [Code]

• SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing
  Jing Gu, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Yilin Wang^, Xin Eric Wang^
  ECCV 2024
  [Paper] [Website] [Code]

• EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing
  Kaizhi Zheng, Xiaotong Chen, Xuehai He, Jing Gu, Linjie Li, Zhengyuan Yang, Kevin Lin, Jianfeng Wang, Lijuan Wang, Xin Eric Wang
  Arxiv
  [Paper] [Website]

• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
  Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao^, Yong Jae Lee^, Jianwei Yang^
  Arxiv
  [Paper] [Website] [Code]

• LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
  Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Jiayang Cheng, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu, Wenpeng Yin
  EMNLP 2024
  [Paper] [Code]

• Muffin or Chihuahua? Challenging Large Vision-Language Models with Multipanel VQA
  Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, Xin Eric Wang
  ACL 2024
  [Paper] [Website] [Code] [Data]

• PHOTOSWAP: Personalized Subject Swapping in Images
  Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang
  NeurIPS 2023
  [paper] [code] [website]

• R2H: Building Multimodal Navigation Helpers that Respond to Help
  Yue Fan, Jing Gu, Kaizhi Zheng, Xin Eric Wang
  EMNLP 2023
  [paper]

• SlugJARVIS: Multimodal Commonsense Knowledge-based Embodied AI for SimBot Challenge
  Jing Gu*, Kaizhi Zheng*, Kaiwen Zhou Yue Fan, Xuehai He Jialu Wang Zonglin Di, Xin Eric Wang
  Alexa Prize SimBot Challenge Proceedings
  [paper]

• JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents
  Kaizhi Zheng*, Jing Gu*, Kaiwen Zhou*, Yue Fan*, Jialu Wang*, Zonglin Di, Xuehai He, Xin Eric Wang
  Preprint
  SoCal NLP 2022
  Winner Model of the Alexa Prize SimBot Public Benchmark Challenge [link]
  [paper]

• Memformer: Memory-Augmented Transformer
  Qingyang Wu, Zhenzhong Lan, Jing Gu, Zhou Yu
  in Proc. of AACL 2022 Findings
  [paper]

• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
  Jing Gu*, Eliana Stefani, Qi Wu, Jesse Thomason, Xin Eric Wang
  in Proc. of ACL 2022
  [paper] [code]

• A Tailored Pre-Training Model for Task-Oriented Dialog Generation
  Jing Gu*, Qingyang Wu*, Chongruo Wu, Weiyan Shi, Zhou Yu
  in Proc. of ACL 2021
  [paper] [code]

• Perception Score, a Learned Metric for Open-ended Text Generation Evaluation
  Jing Gu, Qingyang Wu, Zhou Yu
  in Proc. of AAAI 2021
  [paper]

• Data Annealing for Informal Language Understanding Tasks
  Jing Gu, Zhou Yu
  in Proc. of EMNLP 2020 Findings
  [paper]

• Flow-Aware Structual Model for Conversational Question Generation
  Jing Gu, Mostafa Mirshekari, Zhou Yu, Aaron Sisto
  in Proc. of EACL 2021 [paper] [code]

• ConQuest: Contextual Question Paraphrasing through Answer-Aware Synthetic Question Generation
  Mostafa Mirshekari, Jing Gu, Aaron Sisto
  EMNLP 2021 WNUT Workshop [paper]

Service

Program Committe Member (Reviewer)

• ACL 2021, Sigdial 2021, EMNLP 2021, NeurIPS 2021, ICLR 2022, ACL Rolling Review 2021&2022, NeurIPS 2022, EMNLP 2022, ACL 2023, ICML 2023, IEEE RA-L 23, EMNLP 2023, NeurIPS 2023

Honors & Awards

• Scholar Award of NeurIPS 2023
• Third place ($50,000) in the inaugural Alexa Prize SimBot Challenge! [Media Coverage]
• Winner of the Alexa Prize SimBot Public Benchmark Challenge, May 2022
• Team Leader of SlugJarvis, one of the ten teams selected for Alexa SimBot challenge, ongoing competition, 2021-2023
• Outstanding Graduates Award of Shanghai, June. 2018
• Excellent Graduation Thesis, June. 2018
• Excellent Student Cadre of Shanghai, Dec. 2017

Misc

• Teaching Assitant for Machine Learning, 2021
• Teaching Assitant for Natural Language Processing, 2019
• Teaching Assitant for Machine Dependent Programming, 2020
• Research Scientist in Searchable AI Corp., 2020

One last word

Keep Hungry!