catherine

SEANet. Code for Audio-Visual Target Speaker Extraction with Selective Auditory Attention (TASLP)

SSDQ. Python

SimDINO. [ICML 2025] Official Implementation for SimDINO/SimDINOv2

stable-audio-tools. Generative models for conditional audio generation

Voice-Separation-and-Enhancement. A framework for quick testing and comparing multi-channel speech enhancement and separation methods, such as DSB, MVDR, LCMV, GEVD beamforming and ICA, FastICA, IVA, AuxIVA, OverIVA, ILRMA, FastMNMF.

basic-pitch. A lightweight yet powerful audio-to-MIDI converter with pitch bend detection

ism. Python

hoa. Python

Kimi-Audio. Kimi-Audio, an open-source audio foundation model excelling in audio understanding, generation, and conversation

OmniAudio. [ICML 2025] PyTorch Implementation of "OmniAudio: Generating Spatial Audio from 360-Degree Video"

MMAR. [NeurIPS 2025] Benchmark data and code for MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix

ddsp. DDSP: Differentiable Digital Signal Processing

sfs-python. SFS Toolbox for Python

LAVSS. Code for LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

GWA. Geometric-Wave Acoustic dataset

ss-vq-vae. Self-supervised VQ-VAE for One-Shot Music Style Transfer

control-transfer-diffusion. Repository for the paper "Combining audio control and style transfer using latent diffusion", accepted at ISMIR 2024

Audio-FLAN. Audio-FLAN

seed-vc. zero-shot voice conversion & singing voice conversion, with real-time support

Implementation-of-GelSight. An implementation of Gelsight Wedge.

BNM. code of Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations (CVPR2020 oral)

Versatile-Domain-Adaptation. Code Release for "Minimum Class Confusion for Versatile Domain Adaptation"(ECCV2020)

deepsing. Generating Sentiment-aware Visual Stories using Cross-modal Music Translation

Spatial-AST. 🦇 Encoder of BAT (Learning to Reason about Spatial Sounds with Large Language Models)

ClearerVoice-Studio. An AI-Powered Speech Processing Toolkit and Open Source SOTA Pretrained Models, Supporting Speech Enhancement, Separation, and Target Speaker Extraction, etc.

TCSinger. PyTorch Implementation of TCSinger(EMNLP 2024): Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control

SpatialScaper. Jupyter Notebook

KRT-MLCIL. Python

speechmetrics. A wrapper around speech quality metrics MOSNet, BSSEval, STOI, PESQ, SRMR, SISDR

PanoAVQA. Official repository of PanoAVQA: Grounded Audio-Visual Question Answering in 360° Videos (ICCV 2021)

RealMAN. A description of "RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localization" [NeurIPS 2024]

PianoMotion10M. Code release for PianoMotion10M

Datasets. Poetry-related datasets developed by THUAIPoet (Jiuge) group.

facechain. FaceChain is a deep-learning toolchain for generating your Digital-Twin.

Analytic-continual-learning. This repository will be posting analytic continual learning series, including Analytic Class-Incremental Learning (ACIL), Gaussian Kernel Embedded Analytic Learning (GKEAL), Dual-Stream Analytic Learning (DS-AL), etc.

AV-CIL_ICCV2023. [ICCV 2023] Audio-Visual Class-Incremental Learning

AV-Sepformer. Python

childrenize. Signal processing method to convert adult speech into child-like

SLfM. Official code for the paper: [ICCV2023] Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation

PyTouch. PyTouch is a machine learning library for tactile touch sensing.

TTS. 🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production

VoiceLDM. VoiceLDM: Text-to-Speech with Environmental Context

CREMA-D. Crowd Sourced Emotional Multimodal Actors Dataset (CREMA-D)

FGTD. Face Generation from Textual Description using GANs.

vits. VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

Amphion. Amphion (/æmˈfaɪən/) is a toolkit for Audio, Music, and Speech Generation. Its purpose is to support reproducible research and help junior researchers and engineers get started in the field of audio, music, and speech generation research and development.

Qwen-Audio. The official repo of Qwen-Audio (通义千问-Audio) chat & pretrained large audio language model proposed by Alibaba Cloud.

Real-Time-Voice-Cloning. Clone a voice in 5 seconds to generate arbitrary speech in real-time

VALL-E-X. An open source implementation of Microsoft's VALL-E X zero-shot TTS model. Demo is available in https://plachtaa.github.io/vallex/

encodec. State-of-the-art deep learning based audio codec supporting both mono 24 kHz audio and stereo 48 kHz audio.

vall-e. PyTorch implementation of VALL-E(Zero-Shot Text-To-Speech), Reproduced Demo https://lifeiteng.github.io/valle/index.html

NExT-GPT. Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model

LLaSM. 第一个支持中英文双语语音-文本多模态对话的开源可商用对话模型。便捷的语音输入将大幅改善以文本为输入的大模型的使用体验，同时避免了基于 ASR 解决方案的繁琐流程以及可能引入的错误。

TVLT. PyTorch code for “TVLT: Textless Vision-Language Transformer” (NeurIPS 2022 Oral)

muavic. MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation

SALMONN. SALMONN family: A suite of advanced multi-modal LLMs

ACMR_demo. Python

stereocrw. Code for the Paper: [ECCV2022] Sound Localization by Self-Supervised Time-Delay Estimation

MNBVC. MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。

WavCaps. This reporsitory contains metadata of WavCaps dataset and codes for downstream tasks.

ImageBind. ImageBind One Embedding Space to Bind Them All

LLMZoo. ⚡LLM Zoo is a project that provides data, models, and evaluation benchmark for large language models.⚡

langchain. The agent engineering platform.

TalkLip. Python

GCNet. GCNet, official pytorch implementation of our paper "GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation"

L3DAS23. Official repository supporting the L3DAS23 IEEE ICASSP Grand Challenge

VirtualConductor. 🎶 Music-Driven Conducting Motion Generation (IEEE ICME'21 Best Demo)

s3prl. Self-Supervised Speech Pre-training and Representation Learning Toolkit

Libri-adhoc40. A dataset collected from synchronized ad-hoc microphone arrays

MIntRec. MIntRec: A New Dataset for Multimodal Intent Recognition (ACM MM 2022)

DSTC7-Audio-Visual-Scene-Aware-Dialog-AVSD-Challenge. Python

VoiceSplit. VoiceSplit: Targeted Voice Separation by Speaker-Conditioned Spectrogram

pyroomacoustics. Pyroomacoustics is a package for audio signal processing for indoor applications. It was developed as a fast prototyping platform for beamforming algorithms in indoor scenarios.

voicefilter. Unofficial PyTorch implementation of Google AI's VoiceFilter system

speakerbeam. Jupyter Notebook

move2hear-active-AV-separation. Code and datasets for 'Move2Hear: Active Audio-Visual Source Separation' (ICCV 2021)

learning-audio-visual-dereverberation. Code for paper Learning Audio-Visual Dereverberation

2.5D-Visual-Sound. 2.5D visual sound

BinauralSpeechSynthesis. N/A

FAIR-Play. 2.5D visual sound dataset

waveglow. A Flow-based Generative Network for Speech Synthesis

GACELA. Generative adversarial context encoder for audio inpainting

EasyComDataset. The Easy Communications (EasyCom) dataset is a world-first dataset designed to help mitigate the *cocktail party effect* from an augmented-reality (AR) -motivated multi-sensor egocentric world view.

SoundingEarth. Self-supervised Audiovisual Representation Learning for Remote Sensing Data

audioContextEncoder. A context encoder for audio inpainting

Catch-A-Waveform. Official pytorch implementation of the paper: "Catch-A-Waveform: Learning to Generate Audio from a Single Short Example" (NeurIPS 2021)

ast. Code for the Interspeech 2021 paper "AST: Audio Spectrogram Transformer".

demucs. Code for the paper Hybrid Spectrogram and Waveform Source Separation

speech2gesture. code for training the models from the paper "Learning Individual Styles of Conversational Gestures"

Dancing2Music. Python

TFood. [CVPRW22] Official Implementation of T-Food: "Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval". Accepted at CVPR22 's MULA Workshop.

regnet. Official PyTorch implementation of the TIP paper "Generating Visually Aligned Sound from Videos" and the corresponding Visually Aligned Sound (VAS) dataset.

Wnet. Wnet: Audio-Guided Video Object Segmentation via Wavelet-Based Cross-Modal Denoising Networks

CMBS. cross modal background suppression for audio-visual event localization

meshtalk. Code for MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement

ObjectFolder. ObjectFolder Dataset

AVCA-GZSL. This repository contains the code for our CVPR 2022 paper on "Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language"

DSCMR. Deep Supervised Cross-modal Retrieval (CVPR 2019, PyTorch Code)

headnerf. This repository contains a pytorch implementation of "HeadNeRF: A Real-time NeRF-based Parametric Head Model (CVPR 2022)".

FaceFormer. [CVPR 2022] FaceFormer: Speech-Driven 3D Facial Animation with Transformers

vit-pytorch. Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

SRMRpy. Python implementation of the SRMR toolbox

nara_wpe. Different implementations of "Weighted Prediction Error" for speech dereverberation

SpeechSplit. Unsupervised Speech Decomposition Via Triple Information Bottleneck

voicefixer_main. General Speech Restoration

SkipConvNet. Speech Dereverberation using Fully Convolutional Networks

panns_inference. Python

audio-pretrained-model. A collection of Audio and Speech pre-trained models.

Multimodal-Aerial-Scene-Recognition. Code for <Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition> (ECCV 2020)

pytorch-grad-cam. Advanced AI Explainability for computer vision. Support for CNNs, Vision Transformers, Classification, Object detection, Segmentation, Image similarity and more.

deep-acoustic-analysis. Python

imgaug. Image augmentation for machine learning experiments.

vidaug. Effective Video Augmentation Techniques for Training Convolutional Neural Networks

spatialaudiogen. Spatial Audio Generation

sound-spaces. A first-of-its-kind acoustic simulation platform for audio-visual embodied AI research. It supports training and evaluating multiple tasks and applications.

speech2image. Attempting speech-driven image synthesis.

SAAVN. SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

speechbrain. A PyTorch-based Speech Toolkit

awesome-embodied-vision. Reading list for research topics in embodied vision

image2reverb. [ICCV 2021] Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis.

ACVAE-VC. Jupyter Notebook

autoencoders. implementations of various types of auto-encoders in tensorflow(in progress)

sentence-transformers. State-of-the-Art Embeddings, Retrieval, and Reranking

psla. Code for the TASLP paper "PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation".

animegan2-pytorch. PyTorch implementation of AnimeGANv2

AnimeGANv2. [Open Source]. The improved version of AnimeGAN. Landscape photos/videos to anime

AnimeGAN. A Tensorflow implementation of AnimeGAN for fast photo animation ! This is the Open source of the paper 「AnimeGAN: a novel lightweight GAN for photo animation」, which uses the GAN framwork to transform real-world photos into anime images.

PatchVAE. PyTorch implementation of "PatchVAE: Learning Local Latent Codes for Recognition" to appear in CVPR 2020

av-se. Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

VAE-Torch. Implementation of Variational Auto-Encoder in Torch7

MAE-pytorch. Unofficial PyTorch implementation of Masked Autoencoders Are Scalable Vision Learners

lyrebird-wav2clip. Official implementation of the paper WAV2CLIP: LEARNING ROBUST AUDIO REPRESENTATIONS FROM CLIP

aisle. Official Repository for the paper "No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures", Findings at EMNLP 2020

annotated_deep_learning_paper_implementations. 🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠

ECAPA-TDNN. Unofficial reimplementation of ECAPA-TDNN for speaker recognition (EER=0.86 for Vox1_O when train only in Vox2)

NvEM. [ACM MM 2021 Oral] Official repo of "Neighbor-view Enhanced Model for Vision and Language Navigation"

VAE-ResNet18-PyTorch. A Variational Autoencoder based on the ResNet18-architecture

TorchSSL. A PyTorch-based library for semi-supervised learning (NeurIPS'21)

VisualEchoes. VisualEchoes Dataset (ECCV 2020)

beyond-image-to-depth. Python

ADENET. Code used in the paper "Towards End-to-End Acoustic Localization using Deep Learning: from Audio Signal to Source Position Coordinates"

PyTorch-VAE. A Collection of Variational Autoencoders (VAE) in PyTorch.

InstanceLoc. [CVPR 2021] Instance Localization for Self-supervised Detection Pretraining

MM-DistillNet. PyTorch code for training MM-DistillNet for multimodal knowledge distillation. http://rl.uni-freiburg.de/research/multimodal-distill

KWS-Net. Seeing Wake Words: Audio-visual Keyword Spotting

nerf-pytorch. A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

AD-NeRF. This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".

ml-cpc. Jupyter Notebook

google-research. Google Research

cocon. CoCon: Cooperative Contrastive Learning

General-network-architecture-for-sound-event-localization-and-detection. This repository consists of python code to train sound event localization and detection models.

OpenSound. Various Audio Process Baselines

EIN-SELD. An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection

tal-hmo. Fusional approaches for temporal action localization in untrimmed videos

CVPR2026-Papers-with-Code. CVPR 2026 论文和开源项目合集

seld-dcase2021. Baseline method for sound event localization task of DCASE 2021 challenge

DCASE2020_task1. Code for DCASE 2020 task 1a and task 1b.

AudioCLIP. Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

Awesome-Incremental-Learning. Awesome Incremental Learning

FaceX-Zoo. A PyTorch Toolbox for Face Recognition

Reverberated_WSJ_2MIX. Code to simulate a reverberated, noisy version of the WSJ-2MIX dataset

Cross3D. Code repository for the paper Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks

speech2affective_gestures. This is the official implementation of the paper "Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning".

lyra. A Very Low-Bitrate Codec for Speech Compression

audiomentations. A Python library for audio data augmentation. Useful for making audio ML models work well in the real world, not just in the lab.

L3DAS21. Python

Speech-Emotion-Classification-with-PyTorch. This repository contains PyTorch implementation of 4 different models for classification of emotions of the speech.

mir_eval. Evaluation functions for music/audio information retrieval/signal processing algorithms.

TalkNet-ASD. ACM MM 2021: 'Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection'

fixmatch. A simple method to perform semi-supervised learning with limited data.

MixMatch-pytorch. Code for "MixMatch - A Holistic Approach to Semi-Supervised Learning"

unsupervised-data-augmentation. Unofficial PyTorch Implementation of Unsupervised Data Augmentation.

uda. Unsupervised Data Augmentation (UDA)

MLP-Mixer-pytorch. Unofficial implementation of MLP-Mixer: An all-MLP Architecture for Vision

audioset_tagging_cnn. Python

External-Attention-pytorch. 🍀 Pytorch implementation of various Attention Mechanisms, MLP, Re-parameter, Convolution, which is helpful to further understand papers.⭐⭐⭐

Kaleido-BERT. 💐Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

glow-pytorch. pytorch implementation of openai paper "Glow: Generative Flow with Invertible 1×1 Convolutions"

deep-clustering-1. deep clustering method for single-channel speech separation

DeeperForensics-1.0. [CVPR 2020] A Large-Scale Dataset for Real-World Face Forgery Detection

SpEx_Plus. SpEx+(tied) source code

routing-transformer. Fully featured implementation of Routing Transformer

ai-research-code. Python

DynamicViT. [NeurIPS 2021] [T-PAMI] DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

voxceleb_trainer. In defence of metric learning for speaker recognition

Speaker-Recognition-Demo. A ResNet Speaker Recognition&Verification Demo

CLIP. CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

StyleCLIP. Official Implementation for "StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery" (ICCV 2021 Oral)

gpuRIR. Python library for Room Impulse Response (RIR) simulation with GPU acceleration

scnn. Segment-CNN: A Framework for Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs

SupContrast. PyTorch implementation of "Supervised Contrastive Learning" (and SimCLR incidentally)

SimCLR. PyTorch implementation of SimCLR: A Simple Framework for Contrastive Learning of Visual Representations

pytorchvideo. A deep learning library for video understanding research.

VMZ. VMZ: Model Zoo for Video Modeling

triplet-loss-mnist. Triplet Loss 损失函数

ViLT. Code for the ICML 2021 (long talk) paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

PseudoBinaural_CVPR2021. Codebase for the paper "Visually Informed Binaural Audio Generation without Binaural Audios" (CVPR 2021)

DeepAFx. Third-party audio effects plugins as differentiable layers within deep neural networks.

AVID-CMA. Audio Visual Instance Discrimination with Cross-Modal Agreement

Conv-TasNet. Python