Liangyu Wang

FSDP-Canzona. Bring Muon-style optimizers to PyTorch FSDP training.

Megatron-Canzona. Bring Muon-style optimizers to Megatron-LM training.

Tinytron. A minimal, hackable pre-training stack for GPT-style language models

OpenRollout.

train-large-model-from-scratch. A minimal, hackable pre-training stack for GPT-style language models

picotron. Minimalistic 4D-parallelism distributed training framework for education purpose

Tiny-LLM-Libs.

Tiny-FSDP. Tiny-FSDP, a minimalistic re-implementation of the PyTorch FSDP

Streaming-Dataloader. A memory-efficient streaming data loader designed for LLM pretraining under limited CPU and GPU memory constraints

flashdp. Python

DataMan. Our code for ICLR'25 paper "DataMan: Data Manager for Pre-training Large Language Models".

UTBoost. [ACL'25] UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench

Search-o1. 🔍 Search-o1: Agentic Search-Enhanced Large Reasoning Models [EMNLP 2025]

WebThinker. [NeurIPS 2025] 🌐 WebThinker: Empowering Large Reasoning Models with Deep Research Capability

LLM-Length-Estimation. Python

Tiny-transformers. Python

simple_cuda_kernel. A collection of ultra-simple yet high-performance CUDA kernels.

NanoPD. NanoPD, a minimalistic implementation of the disaggregated PD serve

MetaProfiler. MetaProfiler is a lightweight, structure-agnostic operator-level profiler for PyTorch models that leverages MetaTensor execution to simulate and benchmark individual ops without loading the full model into GPU memory.

AIInfra. AIInfra（AI 基础设施）指AI系统从底层芯片等硬件，到上层软件栈支持AI大模型训练和推理。

AISystem. AISystem 主要是指AI系统，包括AI芯片、AI编译器、AI推理和训练框架等AI全栈底层技术

CoT-Valve. CoT-Valve: Length-Compressible Chain-of-Thought Tuning

LLM-Pruner. [NeurIPS 2023] LLM-Pruner: On the Structural Pruning of Large Language Models. Support Llama-3/3.1, Llama-2, LLaMA, BLOOM, Vicuna, Baichuan, TinyLlama, etc.

learning-to-cache. [NeurIPS 2024] Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching

DeepCache. [CVPR 2024] DeepCache: Accelerating Diffusion Models for Free

WriteHERE. An Open-Source AI Writing Project.

EgoFinger.HCII.SCUT. This repo is about a dataset called EgoFinger, which is maintained by Yichao Huang in SCUT HCII Lab.

Awesome-LLM-System-Papers.

group-invariant-learning. Python

awesome-anti-shortcut-learning. A curated list of out-of-distribution generalization related papers. In progress.

awesome-AI4Drug-papers. A collection of AI for Drug Design related papers and corresponding code sources (in progress).

lazy_emd. Lazy Earth Mover's Distance

FACTS. [ICLR 2025] Implementation of "FACTS: A Factored State-Space Framework For World Modelling"

zo2. ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory

Awesome-ML-SYS-Tutorial. My learning notes for ML SYS.

Fraud-R1. [ACL 2025 Findings] Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements

zo2. ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory [COLM2025]

Flash-Attention-Implementation. Implementation of Flash-Attention (both forward and backward) with PyTorch, CUDA, and Triton

academicpages.github.io. Github Pages template based upon HTML and Markdown for personal, portfolio-based websites.

core_scheduler. CoreScheduler: A High-Performance Scheduler for Large Model Training

adv-ntk. [ICLR 2024] Official repository for "Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach"

CoT-Dataset. Jupyter Notebook

FVLC. Jupyter Notebook

SEAT. Python

SEATv2. Python

FViT. Jupyter Notebook

Tiny-DeepSpeed. Tiny-DeepSpeed, a minimalistic re-implementation of the DeepSpeed library

research-handboook.

long-context-attention. USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

LLM-efficient-learning. Find the most efficient way for a specific large language model to learn a specific task

Tiny-Megatron. Tiny-Megatron, a minimalistic re-implementation of the Megatron library

LLM-efficient-learning. Find the most efficient way for a specific large language model to learn a specific task

Tiny-DeepSpeed. Tiny-DeepSpeed, a minimalistic re-implementation of the DeepSpeed library

textgen. Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private.

LeetCUDA. LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners, 200+ CUDA Kernels, Tensor Cores, HGEMM, FA-2 MMA.

pytorch_influence_functions. This is a PyTorch reimplementation of Influence Functions from the ICML2017 best paper: Understanding Black-box Predictions via Influence Functions by Pang Wei Koh and Percy Liang.

kernel_tuner. Kernel Tuner

CUDA-MatMul-Practice. Cuda

LLM-Trainers. Trainers for LLM Training. Accepts Huggingface models and datasets

system-design-primer. Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

nanoGPT. The simplest, fastest repository for training/finetuning medium-sized GPTs.

Awesome-LLM-Inference. 📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

Finetune_LLMs. Repo for fine-tuning Casual LLMs

matmul_cuda_kernel_tvm. Generate optimized MatMul cuda kernel automatically using tvm auto schedule.

CUDA_gemm. A simple high performance CUDA GEMM implementation.

SpinalResNet. AdderNet ResNet20 for cifar10 written in SpinalHDL

MobileNet-in-FPGA. Generator of verilog description for FPGA MobileNet implementation

hls4ml-tutorial. Tutorial notebooks for hls4ml

cs-self-learning. 计算机自学指南

GPT2. PyTorch Implementation of OpenAI GPT-2

optimization-project. Python

MultimodalWCE.