Zhengxi Liu (刘正曦)

x-transformers. A concise but complete full-attention transformer with a set of promising experimental features from various papers

seed-vc. zero-shot voice conversion & singing voice conversion, with real-time support

LLaSA_training. LLaSA: Scaling Train-time and Inference-time Compute for LLaMA-based Speech Synthesis

DeepSeek-R1.

music_llm. Python

mini_llm. Python

VITA. ✨✨[NeurIPS 2025] VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

moshi. Moshi is a speech-text foundation model and full-duplex spoken dialogue framework. It uses Mimi, a state-of-the-art streaming neural audio codec.

mini-omni. open-source multimodal large language model that can hear, talk while thinking. Featuring real-time end-to-end speech input and streaming audio output conversational capabilities.

WavTokenizer. [ICLR 2025] SOTA discrete acoustic codec models with 40/75 tokens per second for audio language modeling

descript-audio-codec. State-of-the-art audio codec with 90x compression factor. Supports 44.1kHz, 24kHz, and 16kHz mono/stereo audio.

llama-models. Utilities intended for use with Llama models.

llama3. The official Meta Llama 3 GitHub site

fish-speech. SOTA Open Source TTS

videollm-online. VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024)

chameleon. Repository for Meta Chameleon, a mixed-modal early-fusion foundation model from FAIR.

Open-Sora. Open-Sora: Democratizing Efficient Video Production for All

seed-tts-eval. Python

bytedancespeech.github.io. HTML

U-ViT. A PyTorch implementation of the paper "All are Worth Words: A ViT Backbone for Diffusion Models".

SiT. Official PyTorch Implementation of "SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers"

Lumina-T2X. Lumina-T2X is a unified framework for Text to Any Modality Generation

HDTF. the dataset and code for "Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset"

Real-ESRGAN. Real-ESRGAN aims at developing Practical Algorithms for General Image/Video Restoration.

Open-Sora-Plan. This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.

magvit2-pytorch. Implementation of MagViT2 Tokenizer in Pytorch

EMO. Emote Portrait Alive: Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

DiT. Official PyTorch Implementation of "Scalable Diffusion Models with Transformers"

diffae. Official implementation of Diffusion Autoencoders

GPT-SoVITS. 1 min voice data can also be used to train a good TTS model! (few shot voice cloning)

pytorch-vq-vae. PyTorch implementation of VQ-VAE by Aäron van den Oord et al.

bigvsan. Pytorch implementation of BigVSAN

ctc-segmentation. Segment an audio file and obtain utterance alignments. (Python package)

audiocraft. Audiocraft is a library for audio processing and generation with deep learning. It features the state-of-the-art EnCodec audio compressor / tokenizer, along with MusicGen, a simple and controllable music generation LM with textual and melodic conditioning.

flash-attention. Fast and memory-efficient exact attention

WavJourney. WavJourney: Compositional Audio Creation with LLMs

vall-e. PyTorch implementation of VALL-E(Zero-Shot Text-To-Speech), Reproduced Demo https://lifeiteng.github.io/valle/index.html

audio-ai-timeline. A timeline of the latest AI models for audio generation, starting in 2023!

Meta-voicebox. Implementation of Meta-Voicebox : The first generative AI model for speech to generalize across tasks with state-of-the-art performance.

Stepwise_Monotonic_Multihead_Attention. PyTorch Implementation of Stepwise Monotonic Multihead Attention similar to Enhancing Monotonicity for Robust Autoregressive Transformer TTS

Macaw-LLM. Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration

GenAI_LLM_timeline. ChatGPT, GenerativeAI and LLMs Timeline

AcademiCodec. AcademiCodec: An Open Source Audio Codec Model for Academic Research

SpeechGPT. SpeechGPT Series: Speech Large Language Models

WhisperSpeech. An Open Source text-to-speech system built by inverting Whisper.

pytorch-lightning. Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

LLMSurvey. The official GitHub page for the survey paper "A Survey of Large Language Models".

vocalsound. Dataset and baseline code for the VocalSound dataset (ICASSP2022).

StableLM. StableLM: Stability AI Language Models

MOSS. An open-source tool-augmented conversational language model from Fudan University

MiniGPT-4. Open-sourced codes for MiniGPT-4 and MiniGPT-v2 (https://minigpt-4.github.io, https://minigpt-v2.github.io/)

MelNet. Implementation of "MelNet: A Generative Model for Audio in the Frequency Domain"

bark. 🔊 Text-Prompted Generative Audio Model

gpt-2. Code for the paper "Language Models are Unsupervised Multitask Learners"

musiclm-pytorch. Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch

Awesome-LLM. Awesome-LLM: a curated list of Large Language Model

Large-Audio-Models. Keep track of big models in audio domain, including speech, singing, music etc.

large_language_model_training_playbook. An open collection of implementation tips, tricks and resources for training large language models

audiolm-pytorch. Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

libri-light. dataset for lightly supervised training using the librivox audio book recordings. https://librivox.org/.

v-diffusion-pytorch. v objective diffusion inference code for PyTorch.

disco-diffusion. Jupyter Notebook

k-diffusion. Karras et al. (2022) diffusion models for PyTorch

paper-reading. 深度学习经典、新论文逐段精读

AudioLDM. AudioLDM: Generate speech, sound effects, music and beyond, with text.

ChatYuan. ChatYuan: Large Language Model for Dialogue in Chinese and English

audio-diffusion. Apply diffusion models using the new Hugging Face diffusers package to synthesize music instead of images.

vector-quantize-pytorch. Vector (and Scalar) Quantization, in Pytorch

EATS. A pytroch implementation of the EETS: End-to-End Adversarial Text-to-Speech

Speech-Backbones. This is the main repository of open-sourced speech technology by Huawei Noah's Ark Lab.

Text-to-sound-Synthesis. The source code of our paper "Diffsound: discrete diffusion model for text-to-sound generation"

MB-iSTFT-VITS. Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform

improved-diffusion. Release for Improved Denoising Diffusion Probabilistic Models

score_sde_pytorch. PyTorch implementation for Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2021, Oral)

Analytic-DPM. Code for the paper Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (ICLR 2022 Outstanding Paper Award)

iclr2023_stats. ICLR2023 statistics

Poisson_flow. Code for NeurIPS 2022 Paper, "Poisson Flow Generative Models" (PFGM)

cycle-diffusion. [ICCV 2023] A latent space for stochastic diffusion models

NANSY. Python

GON. Gradient Origin Networks - a new type of generative model that is able to quickly learn a latent representation without an encoder

encodec. State-of-the-art deep learning based audio codec supporting both mono 24 kHz audio and stereo 48 kHz audio.

MSMC-TTS. Official Implement of Multi-Stage Multi-Codebook (MSMC) TTS

vdm. Jupyter Notebook

SpeechT5. Unified-Modal Speech-Text Pre-Training for Spoken Language Processing

AdaVocoder. Adaptive Vocoder for Custom Voice

ProDiff. PyTorch Implementation of ProDiff (ACM-MM'22) with a Extremely-Fast diffusion speech synthesis pipeline

interspeech2022.

ddim. Denoising Diffusion Implicit Models

ibot. iBOT :robot:: Image BERT Pre-Training with Online Tokenizer (ICLR 2022)

FastDeploy. High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

ucxpp. C++

cs228-notes. Course notes for CS228: Probabilistic Graphical Models.

audio-diffusion-pytorch. Audio generation using diffusion models, in PyTorch.

stable-diffusion. A latent text-to-image diffusion model

awesome-drones-zh. 无人机资源汇总

VQ-Diffusion. Official implementation of VQ-Diffusion

moco. PyTorch implementation of MoCo: https://arxiv.org/abs/1911.05722

blur-diffusion. Official PyTorch implementation of the paper Progressive Deblurring of Diffusion Models for Coarse-to-Fine Image Synthesis.

LibtorchTutorials. This is a code repository for pytorch c++ (or libtorch) tutorial.

openit. 致力于打造免费无感的翻墙环境

denoising-diffusion-pytorch. Implementation of Denoising Diffusion Probabilistic Model in Pytorch

SYsU-lang. A mini, simple and modular compiler for SYsU/SysY(tiny C). Based on Clang/LLVM/ANTLR4/Bison/Flex.

VQ-Diffusion. Python

AudioMAE. This repo hosts the code and models of "Masked Autoencoders that Listen".

diffusion_models. Minimal standalone example of diffusion model

SummaryOfLoanSuspension. 全国各省市停贷通知汇总

Avocodo-pytorch. Avocodo: Generative Adversarial Network for Artifact-free Vocoder

WaveGrad. Implementation of WaveGrad high-fidelity vocoder from Google Brain in PyTorch.

CLONE.

Comprehensive-E2E-TTS. A Non-Autoregressive End-to-End Text-to-Speech (text-to-wav), supporting a family of SOTA unsupervised duration modelings. This project grows with the research community, aiming to achieve the ultimate E2E-TTS

denoising-diffusion-gan. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs https://arxiv.org/abs/2112.07804

radtts. Provides training, inference and voice conversion recipes for RADTTS and RADTTS++: Flow-based TTS models with Robust Alignment Learning, Diverse Synthesis, and Generative Modeling and Fine-Grained Control over of Low Dimensional (F0 and Energy) Speech Attributes.

tortoise-tts. A multi-voice TTS system trained with an emphasis on quality

wetts. Production First and Production Ready End-to-End Text-to-Speech Toolkit

BigVGAN. Official PyTorch implementation of BigVGAN (ICLR 2023)

Resemblyzer. A python package to analyze and compare voices with deep learning

remote-jobs-in-china. 支持远程办公的中国公司

iSTFTNet-pytorch. iSTFTNet : Fast and Lightweight Mel-spectrogram Vocoder Incorporating Inverse Short-time Fourier Transform

icassp2022.

ReduNet. ReduNet

metaseq. Repo for external large-scale work

conformer. [Unofficial] PyTorch implementation of "Conformer: Convolution-augmented Transformer for Speech Recognition" (INTERSPEECH 2020)

ClariNet. A Pytorch Implementation of ClariNet

run. 润学全球官方指定GITHUB，整理润学宗旨、纲领、理论和各类润之实例；解决为什么润，润去哪里，怎么润三大问题； 并成为新中国人的核心宗教，核心信念。

Recorder. html5 js 录音 mp3 wav ogg webm amr g711a g711u 格式，支持pc和Android、iOS部分Web浏览器、Hybrid App（提供Android iOS App源码）、微信，提供ASR语音识别转文字 H5版语音通话聊天示例 DTMF编码解码

nix-tts. 🐤 Nix-TTS: Lightweight and End-to-end Text-to-Speech via Module-wise Distillation

ssr_eval. Evaluation and Benchmarking of Speech Super-resolution Methods

Learn2Sing2.0. Diffusion and Mutual Information-Based Target Speaker SVS by Learning from Singing Teacher

bddm. BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

HelloSilicon. An introduction to ARM64 assembly on Apple Silicon Macs

hum2song. Hum2Song: Multi-track Polyphonic Music Generation from Voice Melody Transcription with Neural Networks

NeuralSpeech. Python

book-text-to-speech. A book about Text-to-Speech (TTS) in Chinese.

wespeaker. Research and Production Oriented Speaker Verification, Recognition and Diarization Toolkit

IMS-Toucan. Controllable and fast Text-to-Speech for over 7000 languages!

NeuralSVB. Learning the Beauty in Songs: Neural Singing Voice Beautifier; ACL 2022 (Main conference); Official code

academicpages.github.io. Github Pages template based upon HTML and Markdown for personal, portfolio-based websites.

DiffGAN-TTS. PyTorch Implementation of DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

ailab. Python

textlesslib. Library for Textless Spoken Language Processing

NATSpeech. A Non-Autoregressive Text-to-Speech (NAR-TTS) framework, including official PyTorch implementation of PortaSpeech (NeurIPS 2021) and DiffSpeech (AAAI 2022)

Speech-Resources. 语音方向实验室/公司/资源/实习等，欢迎推荐或自荐

Meta-TTS. Official repository of https://doi.org/10.1109/TASLP.2022.3167258. More up-to-date code is in "refactor" branch.

pytorch-revgrad. A minimal pytorch package implementing a gradient reversal layer.

awesome-normalizing-flows. Awesome resources on normalizing flows.

av_hubert. A self-supervised learning framework for audio-visual speech

VAENAR-TTS. PyTorch Implementation of VAENAR-TTS: Variational Auto-Encoder based Non-AutoRegressive Text-to-Speech Synthesis.

flowseq. Generative Flow based Sequence-to-Sequence Toolkit written in Python.

pytorch-softdtw-cuda. Fast CUDA implementation of (differentiable) soft dynamic time warping for PyTorch

notes. Course notes

sandbox. Play time!

nice_pytorch. Flow model - NICE

voicefixer_main. General Speech Restoration

pytorch-flows. PyTorch implementations of algorithms for density estimation

efficient_tts. Pytorch implementation of "Efficienttts: an efficient and high-quality text-to-speech architecture"

cvss. CVSS: A Massively Multilingual Speech-to-Speech Translation Corpus

bert4keras. keras implement of transformers for humans

speech-synthesis-paper. List of speech synthesis papers.

flow. Keras implement of flow-based models

glow-tts. A Generative Flow for Text-to-Speech via Monotonic Alignment Search

Awesome-Diffusion-Models. A collection of resources and papers on Diffusion Models

Catch-A-Waveform. Official pytorch implementation of the paper: "Catch-A-Waveform: Learning to Generate Audio from a Single Short Example" (NeurIPS 2021)

mae. PyTorch implementation of MAE https//arxiv.org/abs/2111.06377

awesome-datascience. :memo: An awesome Data Science repository to learn and apply for real world problems.

chisel-template. A template project for beginning new Chisel work

yatcpu. Yet another toy CPU.

yatcpu-docs. Documentation for YatCPU

opencpop. Opencpop: A High-Quality Open Source Chinese Popular Song Database for Singing Voice Synthesis

DiffSinger. DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (SVS & TTS); AAAI 2022; Official code

phonemizer. Simple text to phones converter for multiple languages

shennong. A Python toolbox for speech features extraction

vocoder-benchmark. A repository for benchmarking neural vocoders by their quality and speed.

chinese-independent-developer. 👩🏿‍💻👨🏾‍💻👩🏼‍💻👨🏽‍💻👩🏻‍💻中国独立开发者项目列表 -- 分享大家都在做什么

silero-models. Silero Models: pre-trained text-to-speech models made embarrassingly simple

YourTTS. YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone

trojan. An unidentifiable mechanism that helps you bypass GFW.

serenity. The Serenity Operating System 🐞

aioway. AI on the way. An auto deep learning pipe dream. An RDBMS approach to deep learning. Declarative, explainable, scalable, optimizable, easy to deploy, all that good stuff.

ICLR2021-OpenReviewData. Crawl & visualize ICLR papers and reviews.

CTCDecoder. Connectionist Temporal Classification (CTC) decoding algorithms: best path, beam search, lexicon search, prefix search, and token passing. Implemented in Python.

python-socketio. Python Socket.IO server and client

socket.io. Bidirectional and low-latency communication for every platform

websocket. Package gorilla/websocket is a fast, well-tested and widely used WebSocket implementation for Go.

PatrickStar. PatrickStar enables Larger, Faster, Greener Pretrained Models for NLP and democratizes AI for everyone.

NUWA. A unified 3D Transformer Pipeline for visual synthesis

cutlass. CUDA Templates and Python DSLs for High-Performance Linear Algebra

Learn-CUDA-Programming. Learn CUDA Programming, published by Packt

Flask-SocketIO. Socket.IO integration for Flask applications.

RAVE. Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

TextGAN-PyTorch. TextGAN is a PyTorch framework for Generative Adversarial Networks (GANs) based text generation models.

redis-py. Redis Python client

transformer-ls. Official PyTorch Implementation of Long-Short Transformer (NeurIPS 2021).

video-bgm-generation. [ACM MM 2021 Best Paper Award] Video Background Music Generation with Controllable Music Transformer

flash-linux0.11-talk. 你管这破玩意叫操作系统源码 — 像小说一样品读 Linux 0.11 核心代码

StyleSpeech. Official implementation of Meta-StyleSpeech and StyleSpeech

PaddleSpeech. Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.

knockknock. 🚪✊Knock Knock: Get notified when your training ends with only two additional lines of code

datasets. 🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

ComputeLibrary. The Compute Library is a set of computer vision and machine learning functions optimised for both Arm CPUs and GPUs using SIMD technologies.

rabbitmq-server. Open source RabbitMQ: core server and tier 1 (built-in) plugins