About

My name is Fuzhao Xue, which is pronounced as Full-Draw Xue in English. Alternatively, you may call me Frio if you prefer. I’m a PhD candidate of HPC-AI under the supervision of Prof. Yang You at National University of Singapore (NUS). I hold an MEng degree from Nanyang Technological University (NTU), where I achieved outstanding academic performance with a perfect GPA of 5.0/5.0 in 2021. During my master’s studies, I was fortunate to be supervised by Prof Eng Siong Chng & Prof Aixin Sun.

Throughout my academic journey, I have had the privilege to collaborate with exceptionally talented scientists in various companies. Previously, I worked as a student researcher at Google Brain, under the guidance of Yi Tay and Mostafa Dehghani. Currently, I am a research intern at NVIDIA GEAR, working alongside Jim Fan and Yuke Zhu. Please check my CV for further information.

My research is generously supported by the Google PhD Fellowship.

I’m actively looking for a few undergrad/master students to work together. Feel free to drop me an email with your CV if you are interested in foundation model research!

Research

My current research mainly focus on Machine Learning, Natural Language Processing, and High Performance Computing. One recent interest is designing algorithm and system to train efficient large language model and other foundation models (e.g. vision, embodied agent). I am always happy to chat about interesting research ideas, and looking for academic collaborations. Please drop me an email if you are interested in collaborating with me.

Selected Projects (all)

Adaptive Computation and Conditional Computation

• OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models [Blog] [Code] [Paper]
  Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou and Yang You

• Adaptive Computation with Elastic Input Sequence [Arxiv] [Code]
  Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, Yang You Accepted at International Conference on Machine Learning (ICML) 2023 (Acceptence rate: 27.9%)

• Go Wider Instead of Deeper [Arxiv] [Code]
  Fuzhao Xue, Ziji Shi, Yuxuan Lou, Yong Liu, Yang You Published at Association for the Advancement of Artificial Intelligence (AAAI) 2022 (Acceptence rate: 15.0%)

Transformer Scaling

• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [Arxiv]
  Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, Yang You Accepted at Neural Information Processing Systems (NeurIPS) 2023 (Acceptence rate: 26.1%)

• A Study on Transformer Configuration and Training Objective [Arxiv] [Blog]
  Fuzhao Xue, Jianghai Chen, Aixin Sun, Xiaozhe Ren, Zangwei Zheng, Xiaoxin He, Yongming Chen, Xin Jiang, Yang You Accepted at International Conference on Machine Learning (ICML) 2023 (Acceptence rate: 27.9%)

AI System for Efficient AI and AI for Efficient AI System

• Sequence Parallelism: Long Sequence Training from System Perspective [Arxiv] [Code] [Video]
  Shenggui Li *, Fuzhao Xue * , Yongbin Li, Yang You Accepted at Association for Computational Linguistics (ACL) 2023 (* indicates equal contribution)

• Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline [Arxiv] [Code] [Blog]
  Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, Yang You Accepted at Neural Information Processing Systems (NeurIPS) 2023 (Acceptence rate: 26.1%)

News

[2023.11]. Awarded Google PhD Fellowship! So many thanks to my wonderful mentors and Google!

[2023.9]. Got two paper (one first-authored paper, i.e. Token-Crisis) accepted to NeurIPS 2023. Cong to Zangwei and myself. So many thanks to my collaborators!

[2023.5]. Got one first-authored paper (equal contribution with Shenggui Li) accepted to ACL 2023. Cong to Shenggui and myself! Thanks to all my collaborators!

[2023.4]. Got two first-authored paper accepted to ICML 2023. Thanks to all my collaborators!

[2023.3]. Got one paper accepted to ICLR 2023 Tiny Track. Congratulations to Liuxiao! It is noteworthy that this paper is extended from one course project. I’m so proud to be the TA of this course and fortunate to work with this team.

[2023.2]. CowClip won the AAAI Distinguished Paper Award. Congratulations to Zangwei and all co-authors!

[2022.11]. Got one paper accepted to AAAI 2023 Oral. Congratulations to Zangwei!

[2022.8]. Got one paper accepted to Artificial Intelligence Review. Congratulations to Jinjie!

[2022.7]. Glad to join Google Brain as a student researcher under the supervision of Yi Tay and Mostafa Dehghani!

Awards

[2023] Google Ph.D. Fellowship

[2023] AAAI 2023 Distinguished Paper Award

[2021] NUS President’s Graduate Fellowship

Professional services

Conference 2024: ICLR (Reviewer), ICML (Reviewer), COLM (Reviewer)

Conference 2023: EMNLP (Reviewer), NeurIPs (Reviewer), CVPR (Reviewer), ACL (PC Member), WWW (Artifacts Reviewer)

Conference 2022: EMNLP (Reviewer), SIGIR (PC Member), ICML (Reviewer)

Journal: TKDE (Reviewer)

Teaching

[2023] CS5260 Neural Networks and Deep Learning II, National University of Singapore

[2022] CS4248 Natural Language Processing, National University of Singapore

Personal information

Personal Hobbies: basketball, fitness and cooking. I also enjoy watching movies and listening to music, although, unfortunately, I’m a terrible singer. Fortunately, I think I have a gift for cooking. Maybe you can say Fuzhao (Frio) is a zero-shot cooking learner. :)