Luowei Zhou

Kimi-K3. Open Frontier Intelligence

AgentENV. AgentENV (AENV) is a distributed platform for running agent environments at scale.

MoonEP. MoonEP: A Perfectly Balanced Expert Parallelism Library via Dynamic Redundant Experts

MetaClaw. 🦞 Just talk to your agent — it learns and EVOLVES 🧬.

AgentsMesh. The AI Agent Workforce Platform. Run a hundred AI coding agents across your own machines — schedule, isolate, and steer them all from one console.

edict. 🏛️ 三省六部制 · OpenClaw Multi-Agent Orchestration System — 9 specialized AI agents with real-time dashboard, model config, and full audit trails

Matrix-Game. Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory

HunyuanWorld-1.0. Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels with Hunyuan3D World Model

ADIEE. Code for ICCV 2025 paper: "ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation"

cline. Autonomous coding agent as an SDK, IDE extension, or CLI assistant.

FunClip. FunASR-powered video transcription, subtitle generation, and LLM-assisted clipping tool with a local Gradio UI.

1d-tokenizer. This repo contains the code for 1D tokenizer and generator

Open-Sora. Open-Sora: Democratizing Efficient Video Production for All

lmms-eval. One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

VLM-R1. Solve Visual Understanding with Reinforced VLMs

ktransformers. A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

gorilla. Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)

multimodal_textbook. [ICCV 2025 Highlight] The official repository for "2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining"

cosmos. NVIDIA Cosmos is an open platform of world models, datasets, and tools that enables developers to build Physical AI for robots, autonomous vehicles, smart infrastructure, and more.

xLAM. xLAM: A Family of Large Action Models to Empower AI Agent Systems

VILA. VILA is a family of state-of-the-art vision language models (VLMs) for diverse multimodal AI tasks across the edge, data center, and cloud.

rho. Repo for Rho-1: Token-level Data Selection & Selective Pretraining of LLMs.

Cosmos-Tokenizer. A suite of image and video neural tokenizers

mle-bench. MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering

Aria. Codebase for Aria - an Open Multimodal Native MoE

ogx. Open GenAI Stack

llama-stack-apps. Agentic components of the Llama Stack APIs

AI-Scientist. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery 🧑‍🔬

MiniCPM-V. A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone

sam2. The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

Spider2-V. [NeurIPS 2024] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

Cradle. The Cradle framework is a first attempt at General Computer Control (GCC). Cradle supports agents to ace any computer task by enabling strong reasoning abilities, self-improvment, and skill curation, in a standardized general environment with minimal requirements.

universal_manipulation_interface. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

Awesome-Video-Diffusion. A curated list of recent diffusion models for video generation, editing, and various other applications.

pinokio. AI Browser

Fooocus. Focus on prompting and generating

SadTalker. [CVPR 2023] SadTalker：Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

audiocraft. Audiocraft is a library for audio processing and generation with deep learning. It features the state-of-the-art EnCodec audio compressor / tokenizer, along with MusicGen, a simple and controllable music generation LM with textual and melodic conditioning.

VGen. Official repo for VGen: a holistic video generation ecosystem for video generation building on diffusion models

gpt-fast. Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

HeyGenClone. A simple and open-source analogue of the HeyGen system

basaran. Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.

CogVLM. a state-of-the-art-level open visual language model | 多模态预训练模型

streaming-llm. [ICLR 2024] Efficient Streaming Language Models with Attention Sinks

triton. Development repository for the Triton language and compiler

ChatDev. ChatDev 2.0: Dev All through LLM-powered Multi-Agent Collaboration

modular. The Modular Platform (includes MAX & Mojo)

Multi-Modality-Arena. Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!

VALL-E-X. An open source implementation of Microsoft's VALL-E X zero-shot TTS model. Demo is available in https://plachtaa.github.io/vallex/

AgentSims. AgentSims is an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in.

RandBox. [ICCV 2023] PyTorch implementation of RandBox

generative_agents. Generative Agents: Interactive Simulacra of Human Behavior

AgentBench. A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)

MetaGPT. 🌟 The Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming

clippinator. AI programming assistant

Wav2Lip. This repository contains the codes of "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild", published at ACM Multimedia 2020. For HD commercial model, please try out Sync Labs

LLM-As-Chatbot. LLM as a Chatbot Service

exllama. A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

ToolBench. [ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.

llama.cpp. LLM inference in C/C++

awesome-langchain. 😎 Awesome list of tools and projects with the awesome LangChain framework

so-vits-svc. SoftVC VITS Singing Voice Conversion

BookGPT. Writes complete books with given paramters, using GPT-3.

mmc4. MultimodalC4 is a multimodal extension of c4 that interleaves millions of images with text.

openai-cookbook. Examples and guides for using the OpenAI API

LLaMA-Adapter. [ICLR 2024] Fine-tuning LLaMA to follow Instructions within 1 Hour and 1.2M Parameters

wolverine. Python

AgentGPT. 🤖 Assemble, configure, and deploy autonomous AI Agents in your browser.

dolly. Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform

GPT-4-LLM. Instruction Tuning with GPT-4

FastChat. An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

openplayground. An LLM playground you can run on your laptop

babyagi. Python

arxiv-bot. Python

chatgpt-well-known-plugin-finder. Checks Alexa's top 1M websites for the presence of OpenAI's new .well-known/ai-plugin.json files

MAD. MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions

EasyLM. Large language models (LLMs) made easy, EasyLM is a one stop solution for pre-training, finetuning, evaluating and serving LLMs in JAX/Flax.

AutoGPT. AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

gpt4all. GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

gpt_academic. 为GPT/GLM等LLM大语言模型提供实用化交互接口，特别优化论文阅读/润色/写作体验，模块化设计，支持自定义快捷按钮&函数插件，支持Python和C++等项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持chatglm3等本地模型。接入通义千问, deepseekcoder, 讯飞星火, 文心一言, llama2, rwkv, claude2, moss等。

chatgpt-retrieval-plugin. The ChatGPT Retrieval Plugin lets you easily find personal or work documents by asking questions in natural language.

llama_index. LlamaIndex is the leading document agent and OCR platform

awesome-totally-open-chatgpt. A list of totally open alternatives to ChatGPT

alpaca-lora. Instruct-tune LLaMA on consumer hardware

OpenChatKit. Python

trl. Train transformer language models with reinforcement learning.

xla. A machine learning compiler for GPUs, CPUs, and ML accelerators

optimate. A collection of libraries to optimise AI model performances

hh-rlhf. Human preference data for "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"

ControlNet. Let us control diffusion models!

langchain. The agent engineering platform.

HIR. Python

summarize.site. Summarize web pages using OpenAI ChatGPT

EVA. EVA Series: Visual Representation Fantasies from BAAI

Open-Assistant. OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.

next.js. The React Framework

big_vision. Official codebase used to develop Vision Transformer, SigLIP, MLP-Mixer, LiT and more.

prompts.chat. f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete privacy.

mind-vis. Code base for MinD-Vis

rl-teacher. Code for Deep RL from Human Preferences [Christiano et al]. Plus a webapp for collecting human feedback

DiffusionDet. [ICCV2023 Best Paper Finalist] PyTorch implementation of DiffusionDet (https://arxiv.org/abs/2211.09788)

jax. Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more

LAVIS. LAVIS - A One-stop Library for Language-Vision Intelligence

TokShift-Transformer. Python

OFA. Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

coyo-dataset. COYO-700M: Large-scale Image-Text Pair Dataset

VideoGPT. Jupyter Notebook

sygil-webui. Stable Diffusion web UI

ml-cvnets. CVNets: A library for training computer vision networks

Prompt-align. [ICCV 2023] Prompt-aligned Gradient for Prompt Tuning

open_clip. An open source implementation of CLIP.

gpt-neox. An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

stable-diffusion. A latent text-to-image diffusion model

gget. 🧬 gget enables efficient querying of genomic reference databases

MSCLIP. Official Code of ECCV 2022 paper MS-CLIP

CodeRL. This is the official code for the paper CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (NeurIPS22).

GroupViT. Official PyTorch implementation of GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR 2022.

MineDojo. Building Open-Ended Embodied Agents with Internet-Scale Knowledge

maskvit.

parti.

t5x. Python

jonbarron.github.io. HTML

BIG-bench. Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

CogVideo. text and image to video generation: CogVideoX (2024) and CogVideo (ICLR 2023)

FlagAI. FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model.

VidIL. Pytorch code for Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

E2FGVI. Official code for "Towards An End-to-End Framework for Flow-Guided Video Inpainting" (CVPR2022)

metaseq. Repo for external large-scale work

ort. Accelerate PyTorch models with ONNX Runtime

latent-diffusion. High-Resolution Image Synthesis with Latent Diffusion Models

dalle-2-preview.

VQ-Diffusion. Python

GLIP. Grounded Language-Image Pre-training

clip-event. Python

how-do-vits-work. (ICLR 2022 Spotlight) Official PyTorch implementation of "How Do Vision Transformers Work?"

mediapipe. Cross-platform, customizable ML solutions for live and streaming media.

s4. Structured state space sequence models

cliport. CLIPort: What and Where Pathways for Robotic Manipulation

BEVT. PyTorch implementation of BEVT (CVPR 2022) https://arxiv.org/abs/2112.01529

svpc. Official implementation of state-aware video procedural captioning (ACM MM 2021)

moment_detr. [NeurIPS 2021] Moment-DETR code and QVHighlights dataset

mae. PyTorch implementation of MAE https//arxiv.org/abs/2111.06377

transformer-ls. Official PyTorch Implementation of Long-Short Transformer (NeurIPS 2021).

SSTAP. Code for our CVPR 2021 Paper "Self-Supervised Learning for Semi-Supervised Temporal Action Proposal".

NUWA. A unified 3D Transformer Pipeline for visual synthesis

SPACH. Python

LOVEU-CVPR2021. Python

DPT. Dense Prediction Transformers

GradCache. Run Effective Large Batch Contrastive Learning Beyond GPU/TPU Memory Constraint

VT-SSum.

FlatCLR. FlatNCE: A Novel Contrastive Representation Learning Objective

Awesome-Zero-Shot-Object-Detection.

decord. An efficient video loader for deep learning with smart shuffling that's super easy to digest

taming-transformers. Taming Transformers for High-Resolution Image Synthesis

Video-Swin-Transformer. This is an official implementation for "Video Swin Transformers".

DALL-E. PyTorch package for the discrete VAE used for DALL·E.

VL-T5. PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)

xcit. Official code Cross-Covariance Image Transformer (XCiT)

vimpac. Python

fastmoe. A fast MoE impl for PyTorch

DataRelease. Data Release for VALUE Benchmark

StarterCode. Starter Code for VALUE benchmark

EvaluationTools. Evaluation code and codalab submission examples for the VALUE benchmark.

ml-contests-conf. ML and DL related contests, competitions and conference challenges.

vince. Video Noise Contrastive Estimation

CvT. This is an official implementation of CvT: Introducing Convolutions to Vision Transformers.

UC2. CVPR 2021 Official Pytorch Code for UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

kinetics-dataset. Shell

simple-amt. A microframework for working with Amazon's Mechanical Turk

dino. PyTorch code for Vision Transformers training with the Self-Supervised learning method DINO

pytorchvideo. A deep learning library for video understanding research.

tesseract. Tesseract Open Source OCR Engine (main repository)

PySceneDetect. :movie_camera: Python and OpenCV-based scene cut/transition detection program & library.

conceptual-12m. Conceptual 12M is a dataset containing (image-URL, caption) pairs collected for vision-and-language pre-training.

universal-computation. Official codebase for Pretrained Transformers as Universal Computation Engines.

Transformer-MM-Explainability. [ICCV 2021- Oral] Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

convit. Code for the Convolutional Vision Transformer (ConViT)

halonet-pytorch. Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

perceiver-pytorch. Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

OWOD. (CVPR 2021 Oral) Open World Object Detection

vissl. VISSL is FAIR's library of extensible, modular and scalable components for SOTA Self-Supervised Learning with images.

ClipBERT. [CVPR 2021 Best Student Paper Honorable Mention, Oral] Official PyTorch code for ClipBERT, an efficient framework for end-to-end learning on image-text and video-text tasks.

CLIP. CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

mmaction2. OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark

audio. Data manipulation and transformation for audio signal processing, powered by PyTorch

Transformer-Explainability. [CVPR 2021] Official PyTorch implementation for Transformer Interpretability Beyond Attention Visualization, a novel method to visualize classifications by Transformer based networks.

pytorch-image-models. The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

Deformable-DETR. Deformable DETR: Deformable Transformers for End-to-End Object Detection.

ig65m-pytorch. PyTorch 3D video classification models pre-trained on 65 million Instagram videos

CV_LTH_Pre-training. [CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

vit-pytorch. Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

digital_video_introduction. A hands-on introduction to video technology: image, video, codec (av1, vp9, h265) and more (ffmpeg encoding). Translations: 🇺🇸 🇨🇳 🇯🇵 🇮🇹 🇰🇷 🇷🇺 🇧🇷 🇪🇸

pytorch-coviar. Compressed Video Action Recognition

faiss. A library for efficient similarity search and clustering of dense vectors.

SlowFast. PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

coot-videotext. COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

vision_transformer. Jupyter Notebook

kaldi. kaldi-asr/kaldi is the official location of the Kaldi project.

YouCook2-Leaderboard. A one-stop shop for YouCook2 info such as leaderboard and recent advances on (cooking) video retrieval and captioning.

Mephisto. A suite of tools for managing crowdsourcing tasks from the inception through to data packaging for research use.