Fuzhao Xue

Fuzhao Xue

PhD Candidate at National University of Singapore.

About

My name is Fuzhao Xue (pronounced “Full-Draw”). My friends call me Frio or Frankie as well. I earned my Ph.D. from the National University of Singapore (NUS) under Prof. Yang You and hold an MEng from Nanyang Technological University (NTU), where I graduated with a perfect 5.0 GPA in 2021 under Prof. Eng Siong Chng and Prof. Aixin Sun. During my PhD, I interned at Google Brain with Yi Tay and Mostafa Dehghani, as well as at NVIDIA GEAR with Jim Fan and Yuke Zhu. My research is supported by the Google PhD Fellowship. For more details, please see my CV.

Research

My current research mainly focus on Machine Learning, Natural Language Processing, and High Performance Computing. One recent interest is designing algorithm and system to train efficient large language model and other foundation models (e.g. vision, embodied agent). I am always happy to chat about interesting research ideas, and looking for academic collaborations. Please drop me an email if you are interested in collaborating with me.

Selected Projects (all)

Efficient Foundation Model Architecture

  • OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models [Blog] [Code] [Paper]
    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou and Yang You Accepted at International Conference on Machine Learning (ICML) 2024 (Acceptence rate: 27.5%)

  • Adaptive Computation with Elastic Input Sequence [Arxiv] [Code]
    Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, Yang You Accepted at International Conference on Machine Learning (ICML) 2023 (Acceptence rate: 27.9%)

Transformer Scaling

  • To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [Arxiv]
    Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, Yang You Accepted at Neural Information Processing Systems (NeurIPS) 2023 (Acceptence rate: 26.1%)

  • A Study on Transformer Configuration and Training Objective [Arxiv] [Blog]
    Fuzhao Xue, Jianghai Chen, Aixin Sun, Xiaozhe Ren, Zangwei Zheng, Xiaoxin He, Yongming Chen, Xin Jiang, Yang You Accepted at International Conference on Machine Learning (ICML) 2023 (Acceptence rate: 27.9%)

Foundation Model Infrastructure

  • LongVILA: Scaling Long-Context Visual Language Models for Long Videos [Code] [Arxiv]
    Fuzhao Xue *, Yukang Chen *, Dacheng Li *, Qinghao Hu *, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, Song Han (* indicates equal contribution)

  • Sequence Parallelism: Long Sequence Training from System Perspective [Arxiv] [Code] [Video]
    Shenggui Li *, Fuzhao Xue * , Yongbin Li, Yang You Accepted at Association for Computational Linguistics (ACL) 2023 (* indicates equal contribution)

Large Language Model Evaluation

  • MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures[Arxiv] [Code] [Homepage]
    Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, Yang You Accepted at Neural Information Processing Systems (NeurIPS) 2024 (Acceptence rate: 25.8%)

News

[2024.9]. Got one paper (MixEval) accepted to NeurIPS 2024. Cong to Jinjie!

[2024.4]. Got one first-authored paper (OpenMoE) accepted to ICML 2024. Thanks to all!

[2023.11]. Awarded by Google PhD Fellowship! So many thanks to my wonderful mentors and Google!

[2023.9]. Got two paper (one first-authored paper, i.e. Token-Crisis) accepted to NeurIPS 2023. Cong to Zangwei and myself. So many thanks to my collaborators!

Awards

[2023] Google Ph.D. Fellowship

[2023] AAAI 2023 Distinguished Paper Award

[2021] NUS President’s Graduate Fellowship

Professional services

Conference 2025: ICLR (Reviewer), AAAI (Reviewer)

Conference 2024: ICLR (Reviewer), ICML (Reviewer), COLM (Reviewer), NeurIPs (Reviewer)

Conference 2023: EMNLP (Reviewer), NeurIPs (Reviewer), ACL (PC Member), WWW (Artifacts Reviewer)

Conference 2022: EMNLP (Reviewer), SIGIR (PC Member), ICML (Reviewer)

Journal: TKDE (Reviewer), ACM Computing Surveys (Reviewer)

Teaching

[2023] CS5260 Neural Networks and Deep Learning II, National University of Singapore

[2022] CS4248 Natural Language Processing, National University of Singapore

Personal information

Personal Hobbies: basketball, fitness and cooking. I also enjoy watching movies and listening to music, although, unfortunately, I’m a terrible singer. Fortunately, I think I have a gift for cooking. Maybe you can say Fuzhao (Frio) is a zero-shot cooking learner. :)