Frank Odom

fft-conv-pytorch. Implementation of 1D, 2D, and 3D FFT convolutions in PyTorch. Much faster than direct convolutions for large kernel sizes.

grouped-query-attention-pytorch. (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (https://arxiv.org/pdf/2305.13245.pdf)

clip-text-decoder. Generate text captions for images from their embeddings.

yet-another-retnet. A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (https://arxiv.org/pdf/2307.08621.pdf)

transformer-from-scratch. Code implementation from my blog post: https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch

soft-mixture-of-experts. PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)

lora-pytorch. Simple but robust implementation of LoRA for PyTorch. Compatible with NLP, CV, and other model types. Strongly typed and tested.

dilated-attention-pytorch. (Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens" (https://arxiv.org/abs/2307.02486)

wnet-unsupervised-image-segmentation. Python

semantle. Fastest 'Semantle' solver this side of the Mississippi.

document-rag. RAG application to answer questions about PDF documents using LLMs.

python-repo-template. Template repo for Python projects, especially those focusing on machine learning and/or deep learning.

byol. PyTorch implementation of BYOL: a fantastically simple method for self-supervised image representation learning with SOTA performance.

simple-bert-pytorch. A very simple BERT implementation in PyTorch, which only depends on PyTorch itself.

metis. Minimalist library for training RL agents in PyTorch. Implements many common training algorithms, with a focus on actor-critic methods. Includes SAC, TD3, PPO, A2C, VPG.

octconv-pytorch. Python

blip-inference. Pretrained BLIP with a similar API to CLIP.

simple-cocotools. A simple, modern alternative to 'pycocotools'.

unipipe. Build batch pipelines in Python that run anywhere.

wordle. Fastest Wordle solver in the West.

gpt-fast. Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

ncsn. Personal repo recreating Noise Contrastive Score Networks (https://arxiv.org/abs/1907.05600v2) for learning purposes.

ddpg-her-pytorch. Implementation of the Hindsight Experience Replay (HER) algorithm using PyTorch. Utilizes Deep Deterministic Policy Gradients for off-policy optimization of RL agents in continuous action spaces.

nanoserp. A tiny, free tool for web search and page scraping leveraging the DuckDuckGo html endpoint. Available as both a CLI and Python package.