Yongyi Zang

genblaze. Genblaze is an open source Python SDK for orchestrating generative AI media pipelines across video, audio, and image providers with built in provenance for every output.

muscriptor. A multi-instrument music transcription model developed by Kyutai and Mirelo.

prose. A new kind of language for a new kind of computer

skills. Skills for Real Engineers. Straight from my .agents directory.

MOSS-TTS-Nano. MOSS-TTS-Nano is an open-source multilingual tiny speech generation model from MOSI.AI and the OpenMOSS team. With only 0.1B parameters, it is designed for realtime speech generation, can run directly on CPU without a GPU, and keeps the deployment stack simple enough for local demos, web serving, and lightweight product integration.

MOSS-TTS. MOSS‑TTS Family is an open‑source speech and sound generation model family from MOSI.AI and the OpenMOSS team. It is designed for high‑fidelity, high‑expressiveness, and complex real‑world scenarios, covering stable long‑form speech, multi‑speaker dialogue, voice/character design, environmental sound effects, and real‑time streaming TTS.

deep-representation-learning-book. Learning Deep Representations of Data Distributions

nnsight. The nnsight package enables interpreting and manipulating the internals of deep learned models.

unipase. Official repository of UniPASE, a SOTA USE model

RCLI. Talk to your Mac, query your docs, no cloud required. On-device voice AI + RAG

trilobyte-lossless-codec. Trilobyte Lossless Codec

slime. slime is an LLM post-training framework for RL Scaling.

heartlib. HeartMuLa Official Repo: The Most Powerful Open-Source Music Generation Model of 2026

e2e. Official JAX implementation of End-to-End Test-Time Training for Long Context

PhysicsLM4. Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

Awesome-ML-SYS-Tutorial. My learning notes for ML SYS.

MiraTTS. A high quality and fast TTS repository

ml-sharp. Sharp Monocular View Synthesis in Less Than a Second

PaCoRe. PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning

sim. Build, deploy, and orchestrate AI agents. Sim is the central intelligence layer for your AI workforce.

windowed-roformer. Official Repository for "Efficient Vocal Source Separation Through Windowed RoFormer"

smule-renaissance. Official Repository of Smule Renaissance, Smule's Vocal Restoration Models

seaweedfs. SeaweedFS is a distributed storage system for object storage (S3), file systems, and Iceberg tables, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.

mellow. small audio language model for reasoning

Syllabus. Synchronized Curriculum Learning for RL Agents

py-cpuinfo. A module for getting CPU info with pure Python

SonicMaster. SonicMaster: Towards Controllable All-in-One Music Restoration and Mastering

AISystem. AISystem 主要是指AI系统，包括AI芯片、AI编译器、AI推理和训练框架等AI全栈底层技术

Wan2.2. Wan: Open and Advanced Large-Scale Video Generative Models

MuQ. Official repository of the paper "MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization".

CSR_Adaptive_Rep. Official Code for Paper: Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation

AntiDeepfake. Project for training SSL-based deepfake speech detector

MultilingualALT. Repo of the paper "Towards Building an End-to-End Multilingual Automatic Lyrics Transcription Model""

genmusic_demo_list. a list of demo websites for automatic music generation research

AReaL. The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.

hnet. H-Net: Hierarchical Network with Dynamic Chunking

mujoco. Multi-Joint dynamics with Contact. A general purpose physics simulator.

aria. Official repository for the paper: Scaling Self-Supervised Representation Learning for Symbolic Piano Performance (ISMIR 2025)

modded-nanogpt. NanoGPT (124M) in 90 seconds

symbolicai. A neurosymbolic perspective on LLMs

aria-midi. Official repository for Aria-MIDI: a MIDI dataset of 1,186,253 transcribed solo-piano recordings.

GDRetriever. Official implementation of the paper - GD-Retriever: Controllable generative text-music retrieval with diffusion models (Accepted at ISMIR25!)

LMCache. LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

1d-tokenizer. This repo contains the code for 1D tokenizer and generator

efficient-spherical-harmonic-evaluation. http://jcgt.org/published/0002/02/06/

SongEval. A song aesthetic evaluation toolkit trained on SongEval.

SonicVerse. SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning

frameless-eval. Lose the Frames: Event-based Metrics for Efficient Music Structure Analysis Evaluation

torch-audiomentations. Fast audio data augmentation in PyTorch. Inspired by audiomentations. Useful for deep learning.

RSTnet. Real-time Speech-Text Foundation Model Toolkit (wip)

MeanFlow. PyTorch implementation of MeanFlow & iMF (one-step generative modeling).

WildFX. Official implementation of WildFX Dataset Generating pipeline.

UCGM. [Preprint] UCGM: Unified Continuous Generative Models

REPA-E. [ICCV 2025] Official implementation of the paper: REPA-E: Unlocking VAE for End-to-End Tuning of Latent Diffusion Transformers

ZeroSep. [NeurIPS 2025] Separate Anything in Audio with Zero Training

Paper2Poster. [NeurIPS 2025] Open-source Multi-agent Poster Generation from Papers

n8n-workflows. all of the workflows of n8n i could find (also from the site itself)

minimp3py. Python bindings for minimp3

verl. verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

DistributionEmbeddings. Jupyter Notebook

arxiv-latex-cleaner. arXiv LaTeX Cleaner: Easily clean the LaTeX code of your paper to submit to arXiv

OpenRLHF-M. An Easy-to-use, Scalable and High-performance RLHF Framework designed for Multimodal Models.

visage. Official implementation of "ViSAGe: Video-to-Spatial AUdio Generation" (ICLR 2025)

LLaDA. Official PyTorch implementation for "Large Language Diffusion Models"

PINN-for-HRTF-upsampling. A physics-informed neural network method for head-related transfer function upsampling

ThunderKittens. Tile primitives for speedy kernels

kleidiai. This repository is a read-only mirror of https://gitlab.arm.com/kleidi/kleidiai

matching-pursuit. This repository contains research and experiments aimed at producing sparse, interpretable representations of audio.

Systems-for-Foundation-Models.

Embodied-AI-Guide. [Lumina具身智能社区] 具身智能技术指南 Embodied-AI-Guide

torchcrepe. Pytorch implementation of the CREPE pitch tracker

DCUnet. Phase-aware speech enchancement with Deep Complex U-Net

eko. Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

sound_field_nn. Python

Dolphin. Dolphin is a multilingual, multitask ASR model jointly trained by DataoceanAI and Tsinghua University.

music_source_separation. Python

r1-aqa. 🤗 R1-AQA Model: mispeech/r1-aqa

sylber. Sylber: Syllabic Embedding Representation of Speech from Raw Audio

LightningDiT. [CVPR 2025 Oral] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models

s1. s1: Simple test-time scaling

audio_caption_metrics. Metrics for evaluating audio caption

torchtitan. A PyTorch native platform for training generative AI models

flash-linear-attention. 🚀 Efficient implementations for emerging model architectures

YuE. YuE: Open Full-song Music Generation Foundation Model, something similar to Suno.ai but open

MP-SENet. Explicit Estimation of Magnitude and Phase Spectra in Parallel for High-Quality Speech Enhancement

stable-ts. Transcription, forced alignment, and audio indexing with OpenAI's Whisper

R3GAN. Code for NeurIPS 2024 paper - The GAN is dead; long live the GAN! A Modern Baseline GAN - by Huang et al.

stable-audio-controlnet. Fine-tune Stable Audio Open with DiT ControlNet.

clarity-template. Clarity: A Minimalist Website Template for AI Research

awesome-lifelong-learning-methods-for-llm. [ACM Computing Surveys 2025] This repository collects awesome survey, resource, and paper for Lifelong Learning with Large Language Models. (Updated Regularly)

Awesome-Multimodal-Next-Token-Prediction. [Survey] Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

Audio-Denoiser-ONNX. Utilizes ONNX Runtime for audio denoising.

VAD_Benchmark. Benchmarking different VAD models on AVA-Speech dataset

ClearerVoice-Studio. An AI-Powered Speech Processing Toolkit and Open Source SOTA Pretrained Models, Supporting Speech Enhancement, Separation, and Target Speaker Extraction, etc.

golf. A DDSP-based neural voice synthesiser.

BigCodec. Official implementation of the paper "BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec"

ssl_speech_restoration_v2. Python

stream-vc. An unofficial PyTorch implementation of the StreamVC(Real-Time Low-Latency Voice Conversion)

pmqd. Perceived Music Quality Dataset

wechatbot-webhook. 轻量、可部署的微信机器人webhook服务，使用http接口收发微信消息, 用它作为个人通知、AIGC 应用或者 coze、n8n等自动化工作流的消息节点

pg. PostgreSQL notes

semantic. Parsing, analyzing, and comparing source code across many languages

BEHAVIOR-1K. BEHAVIOR-1K: a platform for accelerating Embodied AI research. Join our Discord for support: https://discord.gg/bccR5vGFEx

REPA. [ICLR'25 Oral] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think

cgra4ml. An Open Workflow to Build Custom SoCs and run Deep Models at the Edge

SSR-Speech. SSR-Speech: Towards Stable, Safe and Robust Zero-shot Speech Editing and Synthesis

search_rec_ads_papers. Papers on Search, Recommendations, and Ads (搜广推)

ai-by-hand-excel.

MIMO. Official implementation of "MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling"

SOFA. SOFA: Singing-Oriented Forced Aligner

Pake. 🤱🏻 Turn any webpage into a desktop app with one command.

EzAudio. High-quality Text-to-Audio Generation with Efficient Diffusion Transformer

MusicTI_AAAI2024. " Music Style Transfer with Time-Varying Inversion of Diffusion Models"

FoleyCrafter. [IJCV 2026] FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds. AI拟音大师，给你的无声视频添加生动而且同步的音效 😝

FunCodec. FunCodec is a research-oriented toolkit for audio quantization and downstream applications, such as text-to-speech synthesis, music generation et.al.

SEtrain. A training code template for DNN-based speech enhancement.

soundata. Python library for downloading, loading & working with sound datasets

stable-audio-tools. Generative models for conditional audio generation

SimpleTuner. A general fine-tuning kit geared toward image/video/audio diffusion models.

rtfm. Research on Tabular Foundation Models

cambrian. Cambrian-1 is a family of multimodal LLMs with a vision-centric design.

ttt-lm-pytorch. Official PyTorch implementation of Learning to (Learn at Test Time): RNNs with Expressive Hidden States

HPSv2. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis

voyager. 🛰️ An approximate nearest-neighbor search library for Python and Java with a focus on ease of use, simplicity, and deployability.

Cacophony. Inference codebase for "Cacophony: An Improved Contrastive Audio-Text Model". Preprint: https://arxiv.org/abs/2402.06986

WavCraft. Official repo for WavCraft, an AI agent for audio creation and editing

ltfatpy. This is the LTFATPY development repository.

ImHex. 🔍 A Hex Editor for Reverse Engineers, Programmers and people who value their retinas when working at 3 AM.

minGPT. A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training

xtuner. A Next-Generation Training Engine Built for Ultra-Large MoE Models

LLaMA-Adapter. [ICLR 2024] Fine-tuning LLaMA to follow Instructions within 1 Hour and 1.2M Parameters

Video-LLaMA. [EMNLP 2023 Demo] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

SingGraph. Official repository for the paper Singing Voice Graph Modeling for SingFake Detection (Interspeech 2024).

AudioSep. Official implementation of "Separate Anything You Describe"

TRT-SE. An example of a speech enhancement model deployed with TensorRT.

soundctm. Pytorch implementation of SoundCTM

LLM-Codec. The open source code for LLM-Codec

DTTNet-Pytorch. An official implementation of the ICASSP 2024 paper: Dual-Path TFC-TDF UNet for Music Source Separation

ultimatevocalremovergui. GUI for a Vocal Remover that uses Deep Neural Networks.

SimpleDiarization. Simple diarization model

Music-Source-Separation-Training. Repository for training models for music source separation.

dreamgaussian4d. [arXiv 2023] DreamGaussian4D: Generative 4D Gaussian Splatting

AudioLDM. AudioLDM: Generate speech, sound effects, music and beyond, with text.

audio-diffusion. Apply diffusion models using the new Hugging Face diffusers package to synthesize music instead of images.

DiT. Official PyTorch Implementation of "Scalable Diffusion Models with Transformers"

croissant. Croissant is a high-level format for machine learning datasets that brings together four rich layers.

se-scaling. Model configurations for scaling SE models in the paper "Beyond Performance Plateaus: A Comprehensive Study on Scalability in Speech Enhancement"

awesome-jax. JAX - A curated list of resources https://github.com/google/jax

fish-speech. SOTA Open Source TTS

aria-amt. Efficient and robust implementation of seq-to-seq automatic piano transcription.

TaskMatrix. Python

vector-quantize-pytorch. Vector (and Scalar) Quantization, in Pytorch

ddsp_pytorch. Implementation of Differentiable Digital Signal Processing (DDSP) in Pytorch

brouhaha-vad. Predicts the level of noise and reverberation on your audiofiles

ChatTTS. A generative speech model for daily dialogue.

schedule_free. Schedule-Free Optimization in PyTorch

Awesome-LLMs-for-Video-Understanding. 🔥🔥🔥 [IEEE TCSVT] Latest Papers, Codes and Datasets on Vid-LLMs.

AVSpatialAlignment. C++

DiM-DiffusionMamba. The official implementation of DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis

tango. A family of diffusion models for text-to-audio generation.

Awesome-TimeSeries-SpatioTemporal-Diffusion-Model. A survey and paper list of current Diffusion Model for Time Series and SpatioTemporal Data with awesome resources (paper, application, review, survey, etc.).

conditional-flow-matching. TorchCFM: a Conditional Flow Matching library

label-studio. Label Studio is a multi-type data labeling and annotation tool with standardized output format

frechet-audio-distance. A lightweight library for Frechet Audio Distance calculation.

MR-MT3. MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage

basic-pitch-torch. PyTorch version of Spotify's Basic Pitch

timbre-trap. Code for the paper "Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription"

Dynamic3DGaussians. Python

gentle. gentle forced aligner

A-Convolutional-Recurrent-Neural-Network-for-Real-Time-Speech-Enhancement. A minimum unofficial implementation of the "A Convolutional Recurrent Neural Network for Real-Time Speech Enhancement" (CRN) using PyTorch

SceneFake. baseline for SMAD dataset

hilcodec. High fidelity, lightweight, end-to-end, streaming, convolution-based neural audio codec

SpatialScaper. Jupyter Notebook

buddy. BUDDy: Single-Channel Blind Unsupervised Dereverberation with Diffusion Models

minitorch. The full minitorch student suite.

audioseal. Localized watermarking for AI-generated speech audios, with SOTA on robustness and very fast detector

inspect_ai. Inspect: A framework for large language model evaluations

zimtohrli. Jupyter Notebook

gtcrn. The official implementation of GTCRN, an ultra-lightweight SE model.

LLMs_interview_notes. 该仓库主要记录 大模型（LLMs） 算法工程师相关的面试题

LLMs_interview_notes. LLMs interview notes and answers:该仓库主要记录大模型（LLMs）算法工程师相关的面试题和参考答案

demucs. Code for the paper Hybrid Spectrogram and Waveform Source Separation

FreeVC. FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion

Diff-VC. Diffusion Model for Voice Conversion

DiffMorpher. Official Code for DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing (CVPR 2024)

RAVE. Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

AcademiCodec. AcademiCodec: An Open Source Audio Codec Model for Academic Research

EDMSound. Codebase and project page for EDMSound

mamba. Mamba SSM architecture

paper-reading. 深度学习经典、新论文逐段精读

ehrshot-benchmark. A benchmark for few-shot evaluation of foundation models for electronic health records (EHRs)

DawDreamer. Digital Audio Workstation with Python; VST instruments/effects, parameter automation, FAUST, JAX, Warp Markers, and JUCE processors

onnx2tf. A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

StoryTTS. [ICASSP 2024] StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations

streaming. A Data Streaming Library for Efficient Neural Network Training

ffcv. FFCV: Fast Forward Computer Vision (and other ML workloads!)

gcn-tfilm. Modelling black-box audio effects with time-varying feature modulation

music-generation-research. A straightforward collection of Music Generation research resources.

Los-Angeles-MIDI-Dataset. SOTA kilo-scale MIDI dataset for MIR and Music AI purposes

FSD-Dataset. This repository presents FSD dataset for song deepfake detection.