Yuan Gong

whisper-flamingo. Whisper-Flamingo [Interspeech 2024] and mWhisper-Flamingo [IEEE SPL 2025] for Audio-Visual Speech Recognition and Translation

llm_speech_emotion_challenge. Jupyter Notebook

SAIL. SAIL: Search Augmented Instruction Learning

cav-mae. Code and Pretrained Models for ICLR 2023 Paper "Contrastive Audio-Visual Masked Autoencoder".

whisper-at. Code and Pretrained Models for Interspeech 2023 Paper "Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong Audio Event Taggers"

ltu. Code, Dataset, and Pretrained Models for Audio and Speech Large Language Model "Listen, Think, and Understand".

uavm. Code for the IEEE Signal Processing Letters 2022 paper "UAVM: Towards Unifying Audio and Visual Models".

awesome-self-supervised-learning. A curated list of awesome self-supervised methods

pytorch-randaugment. Unofficial PyTorch Reimplementation of RandAugment.

whisper. Robust Speech Recognition via Large-Scale Weak Supervision

paper-reading. 深度学习经典、新论文逐段精读

awesome-grounding. awesome grounding: A curated list of research papers in visual grounding

latent-diffusion. High-Resolution Image Synthesis with Latent Diffusion Models

dataset. Multi30k Dataset

google-scholar-crawler. Crawl google scholar with least code.

crawler-google-scholar. This bot crawls and downloads statistics and pictures from google scholar's researchers.

gscholar-citations-crawler. Crawl all your citations from Google Scholar

kaldi-diar-latte. steps to perform text-based speaker diarization with kaldi toolkit

faiss. A library for efficient similarity search and clustering of dense vectors.

howto100m. Code for the HowTo100M paper

everything_at_once. Official implementation of "Everything at Once - Multi-modal Fusion Transformer for Video Retrieval." CVPR 2022

voice_datasets. 🔊 A comprehensive list of open-source datasets for voice and sound computing (95+ datasets).

dmvr. Python

vocalsound. Dataset and baseline code for the VocalSound dataset (ICASSP2022).

awesome-video-feature-extractor. Video feature extractor in PyTorch.

VGGSound. VGGSound: A Large-scale Audio-Visual Dataset

awesome-audio-visual. A curated list of different papers and datasets in various areas of audio-visual processing

liwc-python. Linguistic Inquiry and Word Count (LIWC) analyzer

video_feature_extractor. Easy to use video deep features extractor

AVLnet. Code for the AVLnet (Interspeech 2021) and Cascaded Multilingual (Interspeech 2021) papers.

scenic. Scenic: A Jax Library for Computer Vision Research and Beyond

byol-pytorch. Usable Implementation of "Bootstrap Your Own Latent" self-supervised learning, from Deepmind, in Pytorch

merlot_reserve. Code release for "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound"

MLIC-KD-WSD. Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection (ACM MM 2018)

ctc-segmentation. Segment an audio file and obtain utterance alignments. (Python package)

espnet. ESPnet with Lightweight Sinc Convolutions and CTC segmentation patches

pase. Problem Agnostic Speech Encoder

ssast. Code for the AAAI 2022 paper "SSAST: Self-Supervised Audio Spectrogram Transformer".

lottery-ticket-hypothesis. A reimplementation of "The Lottery Ticket Hypothesis" (Frankle and Carbin) on MNIST.

Lottery-Ticket-Hypothesis-in-Pytorch. This repository contains a Pytorch implementation of the paper "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Jonathan Frankle and Michael Carbin that can be easily adapted to any model/dataset.

deconstructing-lottery-tickets. Python

PyTorch_CIFAR10. Pretrained TorchVision models on CIFAR10 dataset (with weights)

open_lth. A repository in preparation for open-sourcing lottery ticket hypothesis code.

A-journey-into-Convolutional-Neural-Network-visualization-. A journey into Convolutional Neural Network visualization

BERT-like-is-All-You-Need. The code for our INTERSPEECH 2020 paper - Jointly Fine-Tuning "BERT-like'" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

svcca. Jupyter Notebook

PretrainCNNwithNoData. Python

Audio-Classification. Pytorch code for "Rethinking CNN Models for Audio Classification"

psla. Code for the TASLP paper "PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation".

gopt. Code for the ICASSP 2022 paper "Transformer-Based Multi-Aspect Multi-Granularity Non-native English Speaker Pronunciation Assessment".

pytorch-ssd. MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in Pytorch 1.0 / Pytorch 0.4. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas like CoordConv.

pytorch-cnn-visualizations. Pytorch implementation of convolutional neural network visualization techniques

fcn.berkeleyvision.org. Fully Convolutional Networks for Semantic Segmentation by Jonathan Long*, Evan Shelhamer*, and Trevor Darrell. CVPR 2015 and PAMI 2016.

understanding-transfer-learning. Python

deeplab2. DeepLab2 is a TensorFlow library for deep labeling, aiming to provide a unified and state-of-the-art TensorFlow codebase for dense pixel labeling tasks.

imagenet-sample-images. 1000 images, one per image-net class. For easy visualization/exploration of classes.

perceiver-pytorch. Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

UnsupSeg. Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation (INTERSPEECH 2020)

ShapeTextureDebiasedTraining. Code and models for the paper Shape-Texture Debiased Neural Network Training (ICLR 2021)

augmentation-corruption. This repository provides code for "On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness".

local-global-features-cnn. Code for my Master's Thesis: "The Role of Local Versus Global Features in Convolutional Neural Networks"

model-vs-human. Benchmark your model on out-of-distribution datasets with carefully collected human comparison data (NeurIPS 2021 Oral)

texture-vs-shape. Pre-trained models, data, code & materials from the paper "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness" (ICLR 2019 Oral)

Stylized-ImageNet. Code to create Stylized-ImageNet, a stylized version of standard ImageNet (ICLR 2019 Oral)

torchcrepe. Pytorch implementation of the CREPE pitch tracker

imbalanced-regression. [ICML 2021, Long Talk] Delving into Deep Imbalanced Regression

transformers. 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

kaldi-io-for-python. Python functions for reading kaldi data formats. Useful for rapid prototyping with python.

kaldiio. A pure python module for reading and writing kaldi ark files

voxpopuli. A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation

libri-light. dataset for lightly supervised training using the librivox audio book recordings. https://librivox.org/.

ML-zoo. Python

video-contrastive-learning. Video Contrastive Learning with Global Context, ICCVW 2021

yt-search. JavaScript

BERT-flow. TensorFlow implementation of On the Sentence Embeddings from Pre-trained Language Models (EMNLP 2020)

captum. Model interpretability and understanding for PyTorch

unilm. Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

SpeechSplit. Unsupervised Speech Decomposition Via Triple Information Bottleneck

youtube-dl. Command-line program to download videos from YouTube.com and other video sites

s3prl. Self-Supervised Speech Pre-training and Representation Learning Toolkit

linear-attention-transformer. Transformer based on a variant of attention that is linear complexity in respect to sequence length

linformer. Implementation of Linformer for Pytorch

sam. SAM: Sharpness-Aware Minimization (PyTorch)

knowledge-distillation-pytorch. A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility

Knowledge-Distillation-Zoo. Pytorch implementation of various Knowledge Distillation (KD) methods.

Interspeech2021. This repository describes our reproducible framework for assessing self-supervised representation learning from speech

label-errors. 🛠️ Corrected Test Sets for ImageNet, MNIST, CIFAR, Caltech-256, QuickDraw, IMDB, Amazon Reviews, 20News, and AudioSet

cleanlab. Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Noise-Contrastive-Estimation-NCE-for-pyTorch. Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

CPC_audio. An implementation of the Contrast Predictive Coding (CPC) method to train audio features in an unsupervised fashion.

awesome-knowledge-distillation. Awesome Knowledge Distillation

knowledge-distillation-papers. knowledge distillation papers

Awesome-Visual-Transformer. Collect some papers about transformer with vision. Awesome Transformer with Computer Vision (CV)

Informer2020. The GitHub repository for the paper "Informer" accepted by AAAI 2021.

T2T-ViT. ICCV2021, Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

bertviz. BertViz: Visualize Attention in Transformer Models

espnet. End-to-End Speech Processing Toolkit

axial-attention. Implementation of Axial attention - attending to multi-dimensional data efficiently

pytorch-image-models. The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

ViT-pytorch. Pytorch reimplementation of the Vision Transformer (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale)

einops. Flexible and powerful tensor operations for readable and reliable code (for pytorch, jax, TF and others)

x-transformers. A concise but complete full-attention transformer with a set of promising experimental features from various papers

vit-pytorch. Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

baidudl. This is a multi-thread download tool for pan.baidu.com

awesome-kaldi. This is a list of features, scripts, blogs and resources for better using Kaldi ( http://kaldi-asr.org/ )

PlotNeuralNet. Latex code for making neural networks diagrams

ml-visuals. 🎨 ML Visuals contains figures and templates which you can reuse and customize to improve your scientific writing.

voxceleb-ivector. Voxceleb1 i-vector based speaker recognition system

Speaker-Recognition. This repo contains my attempt to create a Speaker Recognition and Verification system using SideKit-1.3.1

Awesome-Learning-with-Label-Noise. A curated list of resources for Learning with Noisy Labels

kaldi_lm. Old language modeling tool that's used in kaldi

imbalanced-dataset-sampler. A (PyTorch) imbalanced dataset sampler for oversampling low frequent classes and undersampling high frequent ones.

gender-recognition-by-voice. Identify a voice as male or female.

opensmile-python. Python package for openSMILE

surfboard. Novoic's audio feature extraction library

attention-is-all-you-need-pytorch. A PyTorch implementation of the Transformer model in "Attention is All You Need".

bc_learning_sound. Chainer implementation of between-class learning for sound recognition https://arxiv.org/abs/1711.10282

mixup. Implementation of the mixup training method

audioset_tagging_cnn. Python

pytorch. Tensors and Dynamic neural networks in Python with strong GPU acceleration

EfficientNet-PyTorch. A PyTorch implementation of EfficientNet

skynet-ddp-slurm-example. Example of using PyTorch DistributedDataParallel and SLURM on skynet

distributed_tutorial. Python

google-research. Google Research

apex. A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

fairseq. Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Contrastive-Predictive-Coding-PyTorch. Contrastive Predictive Coding for Automatic Speaker Verification

Best-Audio-Classification-Resources-with-Deep-learning. List of articles related to deep learning applied to music

forced-alignment-tools. A collection of links and notes on forced alignment tools

Autoregressive-Predictive-Coding. Autoregressive Predictive Coding: An unsupervised autoregressive model for speech representation learning

Attentive-Filtering-Network. University of Edinbrugh-Johns Hopkins University's system for ASVspoof 2017 Version 2.0 dataset.

ontology. The Audio Set Ontology aims to provide a comprehensive set of categories to describe sound events.

COVID-19-train-audio. COVID-19 Coughs files for training AI models

kaldi-abbr. kaldi name convention note

multichannel-antispoof. Code for SPL paper "Detecting Replay Attacks Using Multi-Channel Audio: A Neural Network-Based Method"

efficient-voice-antispoof. Jupyter Notebook

kaldi. kaldi-asr/kaldi is the official location of the Kaldi project.

sampleCNN-pytorch. Pytorch implementation of "Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms"

examples. A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

SincNet. SincNet is a neural architecture for efficiently processing raw audio samples.

lhvqt. Frontend filterbank learning module with HVQT initialization capabilities.

deep_complex_networks. Implementation related to the Deep Complex Networks

wav2letter. Facebook AI Research's Automatic Speech Recognition Toolkit

pytorch-gradual-warmup-lr. Gradually-Warmup Learning Rate Scheduler for PyTorch

python-compute-eer. Simple Python script to compute equal error rate (EER) for machine learning model evaluation.

pyroomacoustics. Pyroomacoustics is a package for audio signal processing for indoor applications. It was developed as a fast prototyping platform for beamforming algorithms in indoor scenarios.

mic_array. DOA, VAD and KWS for ReSpeaker Microphone Array

Facebook-Interview-Coding-1. 个人整理的Facebook实习面试题目解法，时间范围2016.8-2017.3

BERT-BiLSTM-CRF-NER. Tensorflow solution of NER task Using BiLSTM-CRF model with Google BERT Fine-tuning And private Server services

Name-Entity-Recognition. Lstm-crf,Lattice-CRF,bert-ner及近年ner相关论文follow

ND_replay-attack. Dataset and codes of the audio replay attack research in ND Mobile Computing Lab

sequence_tagging. Named Entity Recognition (LSTM + CRF) - Tensorflow

tf_ner. Simple and Efficient Tensorflow implementations of NER models with tf.estimator and tf.data

CRF. Linear chain conditional random fields are implemented using Numpy and Mxnet/Gluon, and batch training is supported, not limited to training the data one by one.

realtime-adversarial-attack. Code for IJCAI 2019 paper "Real-time Adversarial Attack".

bert. TensorFlow code and pre-trained models for BERT

asvspoof2017. an implement of asvspoof 2017 using pytorch

ReMASC_Exp. Baseline Experiments for ReMASC dataset.

DeepSpeech. DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.

ReMASC. ReMASC: Realistic Replay Attack Corpus for Voice Controlled Systems

Decoupled-Learning-Conditional-GAN. Decoupled Learning for Conditional Adversarial Networks

pix2pix. Image-to-image translation with conditional adversarial nets

wavegan. WaveGAN: Learn to synthesize raw audio with generative adversarial networks

rpgan. RP-GAN: Stable GAN Training with Random Projections

paper-learning-to-pivot. Repository for the paper "Learning to Pivot with Adversarial Networks"

dcgan_tensorflow. Tensorflow implementation of "UNSUPERVISED REPRESENTATION LEARNING WITH DEEP CONVOLUTIONAL GENERATIVE ADVERSARIAL NETWORKS"

text-to-image. Generative Adversarial Text to Image Synthesis / Please Star -->

gan. A 1D toy example of optimizing a generative model using the WGAN-GP model.

pyAudioAnalysis. Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications

word2vec-api. Simple web service providing a word embedding model

cleverhans. An adversarial example library for constructing attacks, building defenses, and benchmarking both

adversarial_BFGS_TensorFlow. Adversarial example creation based on BFGS algorithm implemented under TensorFlow