If you find my projects useful, please star and follow! Everything here comes from my free time, and is released under permissive licenses (e.g. MIT). Send me an email or open an issue if you'd like to discuss a past/current/future project.

PinnedLoading

1. fft-conv-pytorchfft-conv-pytorchPublic

   Implementation of 1D, 2D, and 3D FFT convolutions in PyTorch. Much faster than direct convolutions for large kernel sizes.

   Python 492 60

2. yet-another-retnetyet-another-retnetPublic

   A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (https://arxiv.org/pdf/2307.08621.pdf)

   Python 104 17

3. grouped-query-attention-pytorchgrouped-query-attention-pytorchPublic

   (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (https://arxiv.org/pdf/2305.13245.pdf)

   Python 158 9

4. transformer-from-scratchtransformer-from-scratchPublic

   Code implementation from my blog post: https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch

   Python 92 19

5. clip-text-decoderclip-text-decoderPublic

   Generate text captions for images from their embeddings.

   Python 105 7

6. soft-mixture-of-expertssoft-mixture-of-expertsPublic

   PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)

   Python 71 5