XJ

streamfm. Real-Time Streamable Generative Speech Restoration with Flow Matching

unified-audio. An Open-Source Project to Unify Audio Processing and Generation

moshi-finetune. Python

glm-4-voice-finetune. Python

Awesome-Video-Generation-Post-Training. [TMLR] Video Generation Models: A Survey of Post-Training and Alignment | 🔥 A continuously updated collection of papers, datasets, and benchmarks on post-training and alignment for video generation.

wjk0925.github.io. HTML

Zonos. Zonos-v0.1 is a leading open-weight text-to-speech model trained on more than 200k hours of varied multilingual speech, delivering expressiveness and quality on par with—or even surpassing—top TTS providers.

WSI. Whisper Speaker Identification (WSI), a cutting-edge model for multilingual speaker identification.

F5-TTS. Official code for "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"

slamkit. SlamKit is an open source tool kit for efficient training of SpeechLMs. It was used for "Slamming: Training a Speech Language Model on One GPU in a Day"

audiobox-aesthetics. Unified automatic quality assessment for speech, music, and sound.

stable-codec. A family of state-of-the-art Transformer-based audio codecs for low-bitrate high-quality audio coding.

Machine-Learning-From-Scratch. Implementation of popular ML algorithms from scratch

disentangling_representations. Python

SoloAudio. SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer.

SemantiCodec-inference. Ultra-low bitrate neural audio codec (0.31~1.40 kbps) with a better semantic in the latent space.

MuCodec. Python

Qwen2-Audio. The official repo of Qwen2-Audio chat & pretrained large audio language model proposed by Alibaba Cloud.

moshi. Moshi is a speech-text foundation model and full-duplex spoken dialogue framework. It uses Mimi, a state-of-the-art streaming neural audio codec.

audio-diffusion-pytorch. Audio generation using diffusion models, in PyTorch.

StyleTTS-ZS. StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion

llm-hallucination-survey. Reading list of hallucination in LLMs. Check out our new survey paper: "Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models"

BayLing-Speech. LLaMA-Omni is a low-latency and high-quality end-to-end speech interaction model built upon Llama-3.1-8B-Instruct, aiming to achieve speech capabilities at the GPT-4o level.

MSLDM. Implementation of Multi-Source Music Generation with Latent Diffusion.

slt2024-speechllm-speaker-understanding.

xcodec. AAAI 2025: Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model

BigCodec. Official implementation of the paper "BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec"

audio-ai-hub. The hub for audio AI research: papers, open models, benchmarks & datasets across audio LLMs, speech recognition, TTS, music & audio generation.

Mamba-TasNet. Jupyter Notebook

autoregressive-diffusion-pytorch. Implementation of Autoregressive Diffusion in Pytorch

audio-flamingo. PyTorch implementation of Audio Flamingo: Series of Advanced Audio Understanding Language Models

ssamba. [SLT'24] The official implementation of SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model

Qwen2-Audio. The official repo of Qwen2-Audio chat & pretrained large audio language model proposed by Alibaba Cloud.

Mamba-ASR. ConMamba for Automatic Speech Recognition

ms-swift. Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).

AudioFlamingo. Implementation of the model "AudioFlamingo" from the paper: "Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities"

Awesome-Mamba. Awesome list of papers that extend Mamba to various applications.

mamba. Mamba SSM architecture

mamba-minimal. Simple, minimal implementation of the Mamba SSM in one file of PyTorch.

ICLR-2024-OpenReview-Ratings.

Qwen-Audio. The official repo of Qwen-Audio (通义千问-Audio) chat & pretrained large audio language model proposed by Alibaba Cloud.

Music-Demixing-with-Band-Split-RNN. An unofficial PyTorch implementation of Music Source Separation with Band-split RNN for MDX-23 ("Label Noise" Track)

awesome-large-audio-models. Collection of resources on the applications of Large Language Models (LLMs) in Audio AI.

ltu. Code, Dataset, and Pretrained Models for Audio and Speech Large Language Model "Listen, Think, and Understand".

UniAudio. The Open Source Code of UniAudio

SpeechT5. Unified-Modal Speech-Text Pre-Training for Spoken Language Processing

StyleTTS2. StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

LASS. This repo hosts the code and model of "Separate What You Describe: Language-Queried Audio Source Separation", Interspeech 2022

CLAP. Contrastive Language-Audio Pretraining

clap_curation. Python

CLAP. Learning audio concepts from natural language supervision

SpatialCodec. Implementation of SpatialCodec.

VALL-E-X. An open source implementation of Microsoft's VALL-E X zero-shot TTS model. Demo is available in https://plachtaa.github.io/vallex/

vall-e. An unofficial PyTorch implementation of the audio LM VALL-E

vocos. Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis

SpeechGPT. SpeechGPT Series: Speech Large Language Models

NExT-GPT. Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model

textlesslib. Library for Textless Spoken Language Processing

ODSQA. ODSQA: OPEN-DOMAIN SPOKEN QUESTION ANSWERING DATASET

Awesome-LLM. Awesome-LLM: a curated list of Large Language Model

VGGSound. VGGSound: A Large-scale Audio-Visual Dataset

ml-spatial-librispeech. A large synthetic dataset of spatial audio with multiple labels

llama-cookbook. Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

unilm. Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

llama. Inference code for Llama models

visqol. Perceptual Quality Estimator for speech and audio

audiolm-pytorch. Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

Two-Stage-Polyphonic-Sound-Event-Detection-and-Localization. A two-stage polyphonic sound event detection and localization method for both SED and DOA.

encodec-pytorch. unofficial implementation of the High Fidelity Neural Audio Compression

TAC. transform-average-concatenate (TAC) method for end-to-end microphone permutation and number invariant ad-hoc beamforming.

gpuRIR. Python library for Room Impulse Response (RIR) simulation with GPU acceleration

DCASE2022-data-generator. Data generator for creating synthetic audio mixtures suitable for DCASE Challenge 2022 Task 3

uclser20. Code for the paper "Unsupervised Contrastive Learning of Sound Event Representations", ICASSP 2021.

seld-dcase2023. Baseline method for sound event localization task of DCASE 2023 challenge

vocoder-benchmark. A repository for benchmarking neural vocoders by their quality and speed.

consistency_models. Official repo for consistency models.

stable-diffusion. A latent text-to-image diffusion model

hifi-gan. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

SoundStream. This repository is an implementation of this article: https://arxiv.org/pdf/2107.03312.pdf

NeuralSpeech. Python

audio-diffusion. Apply diffusion models using the new Hugging Face diffusers package to synthesize music instead of images.

diffusers. 🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.

Awesome-Diffusion-Models. A collection of resources and papers on Diffusion Models

FullySpikingVAE. Official implementation of Fully Spiking Variational Autoencoder [AAAI2022]

espnet. End-to-End Speech Processing Toolkit

s3prl. Self-Supervised Speech Pre-training and Representation Learning Toolkit

avalanche. Avalanche: an End-to-End Library for Continual Learning based on PyTorch.

Knowledge-Distillation-Zoo. Pytorch implementation of various Knowledge Distillation (KD) methods.

SPQ. Self-supervised Product Quantization for Deep Unsupervised Image Retrieval - ICCV2021

pyloudnorm. Flexible audio loudness meter in Python with implementation of ITU-R BS.1770-4 loudness algorithm

Conv-TasNet. Python

Discriminator-Constrained-Optimal-Transport-Network. Python

sound-separation. Python

cssl_sound. Python

barlowtwins. PyTorch implementation of Barlow Twins.

pytorch-metric-learning. The easiest way to use deep metric learning in your application. Modular, flexible, and extensible. Written in PyTorch.

DANN. pytorch implementation of Domain-Adversarial Training of Neural Networks

awesome-domain-adaptation. A collection of AWESOME things about domain adaptation

speechbrain. A PyTorch-based Speech Toolkit

BBAVectors-Oriented-Object-Detection. [WACV2021] Oriented Object Detection in Aerial Images with Box Boundary-Aware Vectors

pytorch-vqvae. Vector Quantized VAEs - PyTorch Implementation

vector-quantize-pytorch. Vector (and Scalar) Quantization, in Pytorch

linux. Linux kernel source tree

denoiser. Real Time Speech Enhancement in the Waveform Domain (Interspeech 2020)We provide a PyTorch implementation of the paper Real Time Speech Enhancement in the Waveform Domain. In which, we present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is optimized on both time and frequency domains, using multiple loss functions. Empirical evidence shows that it is capable of removing various kinds of background noise including stationary and non-stationary noises, as well as room reverb. Additionally, we suggest a set of data augmentation techniques applied directly on the raw waveform which further improve model performance and its generalization abilities.

pytorch. Tensors and Dynamic neural networks in Python with strong GPU acceleration

ECE391-Kernel. C