Yi (Jerry) Li

oh-my-codex. OmX - Oh My codeX: Your codex is not alone. Add hooks, agent teams, HUDs, and so much more.

Attention-Residuals.

tinyworlds. A minimal implementation of DeepMind's Genie world model

reasoning-from-scratch. Implement a reasoning LLM in PyTorch from scratch, step by step

vggt. [CVPR 2025 Best Paper Award] VGGT: Visual Geometry Grounded Transformer

gemini-cli. An open-source AI agent that brings the power of Gemini directly into your terminal.

SpatialLM. [NeurIPS 2025] SpatialLM: Training Large Language Models for Structured Indoor Modeling

genesis-world. Simulation platform for general-purpose robotics & embodied AI learning.

DeepSeek-VL2. DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding

hymba. Python

granite-snack-cookbook. Granite Snack Cookbook -- easily consumable recipes (python notebooks) that showcase the capabilities of the Granite models

Cosmos-Tokenizer. A suite of image and video neural tokenizers

papers_we_read. Summaries for exciting works in the field of Deep Learning.

ChatDev. ChatDev 2.0: Dev All through LLM-powered Multi-Agent Collaboration

LLaVA-NeXT. Python

EgoVLPv2. Code release for "EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone" [ICCV, 2023]

gpt_paper_assistant. GPT4 based personalized ArXiv paper assistant bot

syn-rep-learn. Learning from synthetic data - code and models

grok-1. Grok open release

Segment-Everything-Everywhere-All-At-Once. [NeurIPS 2023] Official implementation of the paper "Segment Everything Everywhere All at Once"

NExT-GPT. Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model

DallEval. DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (ICCV 2023)

al-folio. A beautiful, simple, clean, and responsive Jekyll theme for academics

Academic-project-page-template. A project page template for academic papers. Demo at https://eliahuhorwitz.github.io/Academic-project-page-template/

time-diffusion. Official code repo for "Editing Implicit Assumptions in Text-to-Image Diffusion Models"

visprog. Official code for VisProg (CVPR 2023 Best Paper!)

ControlNet-v1-1-nightly. Nightly release of ControlNet 1.1

debias-gensynth. Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Sets

LLaVA. [NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

TPT. Test-time Prompt Tuning (TPT) for zero-shot generalization in vision-language models (NeurIPS 2022))

textgen. Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private.

ControlNet. Let us control diffusion models!

optimum-habana. Easy and lightning fast training of 🤗 Transformers on Habana Gaudi processor (HPU)

debias_vl. Code for Debiasing Vision-Language Models via Biased Prompts

ChatGPT. Reverse engineered ChatGPT API

instruct-pix2pix. Python

Crisscrossed-Captions. Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

multimodal. TorchMultimodal is a PyTorch library for training state-of-the-art multimodal multi-task models at scale.

LAVIS. LAVIS - A One-stop Library for Language-Vision Intelligence

LM_bias. [ICML 2021] Towards Understanding and Mitigating Social Biases in Language Models

nanoGPT. The simplest, fastest repository for training/finetuning medium-sized GPTs.

Networks-Beyond-Attention. A compilation of network architectures for vision and others without usage of self-attention mechanism

Awesome-Diffusion-Models. A collection of resources and papers on Diffusion Models

MeMViT. Code Release for MeMViT Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition, CVPR 2022

mmengine. OpenMMLab Foundational Library for Training Deep Learning Models

kinetics-dataset. Shell

awesome-tips.

DeltaCNN. DeltaCNN End-to-End CNN Inference of Sparse Frame Differences in Videos

mvit. Code Release for MViTv2 on Image Recognition.

ConvNeXt. Code release for ConvNeXt model

yt-dlp. A feature-rich command-line audio/video downloader

SparseConvNet. Submanifold sparse convolutional networks

gdown. Google Drive public file downloader when curl/wget fails.

composer. Supercharge Your Model Training

MTTR. Python

math. The MATH Dataset (NeurIPS 2021)

visualbert. Code for the paper "VisualBERT: A Simple and Performant Baseline for Vision and Language"

Revisit-MMT. Python

mdetr. Python

DALLE-pytorch. Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

CLIP-ViL. [ICLR 2022] code for "How Much Can CLIP Benefit Vision-and-Language Tasks?" https://arxiv.org/abs/2107.06383

wit. WIT (Wikipedia-based Image Text) Dataset is a large multimodal multilingual dataset comprising 37M+ image-text sets with 11M+ unique images across 100+ languages.

flores. Facebook Low Resource (FLoRes) MT Benchmark

CLIP. CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

miniClip. Python

CondensedMovies. Story-Based Retrieval with Contextual Embeddings. Largest freely available movie video dataset. [ACCV'20]

XLM. PyTorch original implementation of Cross-lingual Language Model Pretraining.

pytorchvideo. A deep learning library for video understanding research.

fairseq. Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

pytorch-lightning. Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

smart_open. Utils for streaming large files (S3, HDFS, gzip, bz2...)

VMZ. VMZ: Model Zoo for Video Modeling

SlowFast. PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

mmaction. An open-source toolbox for action understanding based on PyTorch

ar-cutpaste. Cut and paste your surroundings using AR

awesome. 😎 Awesome lists about all kinds of interesting topics

CATER. CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning

temporal-shift-module. [ICCV 2019] TSM: Temporal Shift Module for Efficient Video Understanding

detectron2. Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.

natural-adv-examples. A Harder ImageNet Test Set (CVPR 2021)

Optical-Flow-GPU-Docker. Compute dense optical flow using TV-L1 algorithm with NVIDIA GPU acceleration.

PyVideoResearch. A repository of common methods, datasets, and tasks for video research

CVPR2019. Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.

nvvl. A library that uses hardware acceleration to load sequences of video frames to facilitate machine learning training

PyAV. Pythonic bindings for FFmpeg's libraries.

football. Check out the new game server:

texture-vs-shape. Pre-trained models, data, code & materials from the paper "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness" (ICLR 2019 Oral)

Domain-generalization. All about domain generalization

AlphaPose. Real-Time and Accurate Full-Body Multi-Person Pose Estimation&Tracking System

DetectAndTrack. The implementation of an algorithm presented in the CVPR18 paper: "Detect-and-Track: Efficient Pose Estimation in Videos"

awesome-action-recognition. A curated list of action recognition and related area resources

Action_Recognition_Zoo. Codes for popular action recognition models, verified on the something-something data set.

dense_flow. OpenCV Implementation of different optical flow algorithms

pytorch-rotationnet. Python

temporal-fields. Code for training temporal fully-connected CRF models in Torch

Conference-Acceptance-Rate. Acceptance rates for the major AI conferences

manim. Animation engine for explanatory math videos

Non-local_pytorch. Implementation of Non-local Block.

towards-realistic-predictors. pytorch implementation for paper, towards realistic predictors

Detectron.pytorch. A pytorch implementation of Detectron. Both training from scratch and inferring directly from pretrained Detectron weights are available.

two-stream-action-recognition. Using two stream architecture to implement a classic action recognition method on UCF101 dataset

kinetics_i3d_pytorch. Inflated i3d network with inception backbone, weights transfered from tensorflow

3D-ResNets-PyTorch. 3D ResNets for Action Recognition (CVPR 2018)

TorchFusion. A modern deep learning framework built to accelerate research and development of AI systems

skorch. A scikit-learn compatible neural network library that wraps PyTorch

faster-rcnn.pytorch. A faster pytorch implementation of faster r-cnn

tnt. A lightweight library for PyTorch training tools and utilities

pytorch-custom-dataset-examples. Some custom dataset examples for PyTorch

pretrained-models.pytorch. Pretrained ConvNets for pytorch: NASNet, ResNeXt, ResNet, InceptionV4, InceptionResnetV2, Xception, DPN, etc.

tensorboardX. tensorboard for pytorch (and chainer, mxnet, numpy, ...)

pytorch-playground. Base pretrained models and datasets in pytorch (MNIST, SVHN, CIFAR10, CIFAR100, STL10, AlexNet, VGG16, VGG19, ResNet, Inception, SqueezeNet)

visdom. A flexible tool for creating, organizing, and sharing visualizations of live, rich data. Supports Torch and Numpy https://visdom.dev

pytorch-tutorial. PyTorch Tutorial for Deep Learning Researchers

pytorch-meta-optimizer. A PyTorch implementation of Learning to learn by gradient descent by gradient descent

pytorch. Tensors and Dynamic neural networks in Python with strong GPU acceleration