Lirui (Leroy) Wang

HMA. Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression

open-pi-zero. Re-implementation of pi0 vision-language-action (VLA) model from Physical Intelligence

itps. Python

GenSim2. Python

HPT. Heterogeneous Pre-trained Transformer (HPT) as Scalable Policy Learner.

nanoGPT. The simplest, fastest repository for training/finetuning medium-sized GPTs.

t3. Repository for Transferable Tactile Transformers (T3)

prismatic-vlms. A flexible and efficient codebase for training visually-conditioned language models (VLMs)

SimplerEnv. Evaluating and reproducing real-world robot manipulation policies (e.g., RT-1, RT-1-X, Octo) in simulation under common setups (e.g., Google Robot, WidowX+Bridge) (CoRL 2024)

llama3. The official Meta Llama 3 GitHub site

GraspTrajOpt. Trajectory optimization for grasping

gpt-fast. Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

generative-models. Generative Models by Stability AI

AdaptSim. Code Release for AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer

f3rm. F3RM: Feature Fields for Robotic Manipulation. Official repo for the paper "Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation" (CoRL 2023).

Fleet-Tools. Tool-use Robotic Benchmark built with Drake Simulation

threednel. Python

Tracking-Anything-with-DEVA. [ICCV 2023] Tracking Anything with Decoupled Video Segmentation

loopquest. A Production Tool for Embodied AI

fairscale. PyTorch extensions for high performance and large scale training.

codellama. Inference code for CodeLlama models

Dynamic3DGaussians. Python

world-creator. LLM-based CLI utility for simulation worlds creation.

GenSim. Generating Robotic Simulation Tasks via Large Language Models

BundleSDF. [CVPR 2023] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

repro. Simple area to test out concepts or try to reproduce isolated issues.

aloha. Python

segment-anything. The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

open_clip. An open source implementation of CLIP.

diffusion_policy. [RSS 2023] Diffusion Policy Visuomotor Policy Learning via Action Diffusion

avatar_simulation. Python

drake-ros. Experimental prototyping (for now)

stable-diffusion. A latent text-to-image diffusion model

scenic. Scenic: A Jax Library for Computer Vision Research and Beyond

git-re-basin. Code release for "Git Re-Basin: Merging Models modulo Permutation Symmetries"

grasp_diffusion. Pytorch implementation of diffusion models on Lie Groups for 6D grasp pose generation https://sites.google.com/view/se3dif/home

NGDF. Neural Grasp Distance Fields for Robot Manipulation

point-e. Point cloud diffusion for 3D model synthesis

Assemble-Them-All. [SIGGRAPH Asia 2022] Assemble Them All: Physics-Based Planning for Generalizable Assembly by Disassembly

polyform. Tools for working with raw Polycam data, including export for NeRF and other formats.

robotics_utilities. Python

Dec-SSL. Understanding Self-Supervised Learning in a non-IID Setting

quasistatic_simulator_cpp. C++

ModernRobotics. Modern Robotics: Mechanics, Planning, and Control Code Library --- The primary purpose of the provided software is to be easy to read and educational, reinforcing the concepts in the book. The code is optimized neither for efficiency nor robustness.

ngp_pl. Instant-ngp in pytorch+cuda trained with pytorch-lightning (high quality with high speed, with only few lines of legible code)

meshcat-python. WebGL-based 3D visualizer for Python

manipulation. Course notes for MIT manipulation class

OmniIsaacGymEnvs. Reinforcement Learning Environments for Omniverse Isaac Gym

furniture. IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks

hidapi. A Simple cross-platform library for communicating with HID devices

instant-ngp. Instant neural graphics primitives: lightning fast NeRF and more

handover-sim. A simulation environment and benchmark for human-to-robot object handovers

eirli. An Empirical Investigation of Representation Learning for Imitation (EIRLI), NeurIPS'21

google-research. Google Research

planning_through_contact. Python

colmap. COLMAP - Structure-from-Motion and Multi-View Stereo

labelme. Image annotation with Python. Supports polygon, rectangle, circle, line, point, and AI-assisted annotation.

clip-retrieval. Easily compute clip embeddings and build a clip retrieval system with them

IsaacGymEnvs. Isaac Gym Reinforcement Learning Environments

nerf-supervision-public. Python

Clip-Seg. Open-Text Semantics Segmentation for Robotics Manipulation

stretch_ros. ROS packages for the Stretch mobile manipulators from Hello Robot Inc.

Federated-Learning-PyTorch. Implementation of Communication-Efficient Learning of Deep Networks from Decentralized Data

SimCLR. A PyTorch implementation of SimCLR based on ICML 2020 paper "A Simple Framework for Contrastive Learning of Visual Representations"

ycb_renderer. C++

cliport. CLIPort: What and Where Pathways for Robotic Manipulation

BlenderProc. A procedural Blender pipeline for photorealistic training image generation

robot-plan-runner. C++

vissl. VISSL is FAIR's library of extensible, modular and scalable components for SOTA Self-Supervised Learning with images.

awesome-self-supervised-learning. A curated list of awesome self-supervised methods

CLIP. CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

ravens. Train robotic agents to learn pick and place with deep learning for vision-based manipulation in PyBullet. Transporter Nets, CoRL 2020.

HCG. Cluttered-Scene 6D Grasping with Latent Plans

GA-DDPG. Dexterous Grasping Policy from Point Clouds

OMG-Planner. An Optimization-based Motion and Grasp Planner

learning_from_play. Jupyter Notebook

blender_server. Python

vgn. Real-time 6 DOF grasp detection in clutter.

bullet3. Bullet Physics SDK: real-time collision detection and multi-physics simulation for VR, games, visual effects, robotics, machine learning etc.

drake. Model-based design and verification for robotics.

LearningToLearn. Collection of algorithms to learn loss and reward functions via gradient-based bi-level optimization.

PoseCNN-PyTorch. PyTorch implementation of the PoseCNN framework

torchdiffeq. Differentiable ODE solvers with full GPU support and O(1)-memory backpropagation.

caspr. Code for NeurIPS 2020 paper "CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations"

DeepIM-PyTorch. PyTorch implementation of the DeepIM framework

higher. higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.

vision-based-robotic-grasping. Related papers and codes for vision-based robotic grasping

MinkowskiEngine. Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors

pytorch_6dof-graspnet. Python

pybullet-planning. PyBullet Planning

PyTorch-RL. PyTorch implementation of Deep Reinforcement Learning: Policy Gradient methods (TRPO, PPO, A2C) and Generative Adversarial Imitation Learning (GAIL). Fast Fisher vector product TRPO.

Metaworld. Collections of robotics environments geared towards benchmarking multi-task and meta reinforcement learning

pytorch-soft-actor-critic. PyTorch implementation of soft actor critic

robustness. A library for experimenting with, training and evaluating neural networks, with a focus on adversarial robustness.

learning-to-reweight-examples. PyTorch Implementation of the paper Learning to Reweight Examples for Robust Deep Learning

deepq-grasping. Python

PointNetGPD. PointNetGPD is an end-to-end grasp evaluation model to address the challenging problem of localizing robot grasp configurations directly from the point cloud.

robotPose. 7-dof Robot Arm Kinematics and Grasp Utilities

graspit-pkgs. Collection of packages related to GraspIt!

mx-DeepIM. Deep Iterative Matching for 6D Pose Estimation

LabelFusion. LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

PoseCNN. A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

robopy. Robopy is a python port for Robotics Toolbox in Matlab created by Peter Corke

monodepth. Unsupervised single image depth prediction with CNNs

demon. DeMoN: Depth and Motion Network

flownet2-tf. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

redtail. Perception and AI components for autonomous mobile robotics.

RoboRTS. An open source software stack for Real-Time Strategy research on mobile robots

robo-miscs. HTML