Repository files navigation

📚 LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners 🐑

📚LeetCUDA: It includesTensor/CUDA Cores, TF32/F16/BF16/F8,📖200+ CUDA Kernels🔥 with PyTorch,📖100+ LLM/CUDA🔥 blogs,📖HGEMM⚡️ which can achieve98%~100% TFLOPS ofcuBLAS, and📖flash-attn⚡️ using Tensor Cores with pure MMA PTX.♥️ Please consider to leave a ⭐️ Star to support me, my bro ~♥️

@misc{LeetCUDA@2025,title={LeetCUDA: A Modern CUDA Learn Notes with PyTorch for Beginners},url={https://github.com/xlite-dev/LeetCUDA.git},note={Open-source software available at https://github.com/xlite-dev/LeetCUDA.git},author={DefTruth and Many Others},year={2025}}

• [2025-08-18]:🤗cache-dit is released! 🤗A PyTorch-native Inference Engine with Hybrid Cache Acceleration and Parallelism for DiTs. Feel free to take a try!

• [2025-01-08]:🤖ffpa-attn is released! Yet another Faster Flash Prefill Attention with O(1)🎉SRAM complexity for large headdim,1.8x~3x↑🎉 vs SDPA EA:📈L20 ~1.9x↑🎉,📈A30 ~1.8x↑🎉,📈4090 ~2.1x↑🎉.

• [2024-12-02]:⚡️HGEMM is released! Write HGEMM from scratch using Tensor Cores withWMMA, MMA and CuTe API, achieve peak🎉 performance.

• 📖 HGEMM-MMA 🎉🎉
• 📖 FlashAttention-MMA 🎉🎉
  • 📚 Split KV (Basic, FA-1)
  • 📚 Split Q (Faster, FA-2)
  • 📚 Split Q + Shared KV
  • 📚 Split Q + Shared QKV
  • 📚 Split Q + QK Tiling
  • 📚 Split Q + QKV Tiling
• 📖 200+ CUDA Kernels 🔥🔥
  • 📚 Easy ⭐️
  • 📚 Medium ⭐️⭐️
  • 📚 Hard ⭐️⭐️⭐️
  • 📚 Hard+ ⭐️⭐️⭐️⭐️
  • 📚 Hard++ ⭐⭐⭐️⭐️⭐️
  • 📚 Triton ⭐⭐⭐️
  • 📚 CUTLASS ⭐⭐⭐️
• 📖 100+ LLM/CUDA Blogs 🔥
• 📖 How to Contribute 👀👇

Currently, on NVIDIA L20, RTX 4090 and RTX 3080 Laptop, compared with cuBLAS's default Tensor Cores algorithm, theHGEMM (WMMA/MMA/CuTe) in this repo (blue🔵) can achieve98%~100% of its (orange🟠) performance. Please checktoy-hgemm library⚡️⚡️ orHGEMM⚡️⚡️ repo for more details.

I have also implementedFlashAttention-2 using pure MMA PTX instructions, which supports features such as Multi-Stages, Tile MMA, Tile Warp, Shared KV SMEM,Fully Shared QKV SMEM,Prefetch Q s2r,Prefetch K/V g2s,QKV Fine-grained Tiling, Collective Store, etc. Please refer toflash-attn⚡️⚡️ for more details.

Currently, for small-scale attention(B<=4, H <=48, SeqLen <= 8192, D <= 64) it can run faster than FA2/SDPA on some Devices. For example, on NVIDIA RTX 3080 Laptop,📚 Split Q + Fully Shared QKV SMEM method can achieve55 TFLOPS (D=64) that almost~1.5x 🎉 faster than FA2. On NVIDIA L20, 🤖ffpa-attn method can achieve104 TFLOPS (D=512) that almost~1.8x 🎉 faster than SDPA (EFFICIENT ATTENTION). However, for large-scale attention, there remains a performance gap. Stay tuned for updates ~ (MMA Acc F16/F32, softmax Acc F32 vs FA2 MMA/softmax Acc F32, 👇Benchmark)

TheSplit KV andSplit Q implementations have been carried out inflash-attn⚡️⚡️ for performance comparison. TheSplit KV method, which involves splitting all QKV across MMA (Warps), is slower thanSplit Q method, which splitting Q across MMA(Warps) and keep access KV for all MMA(Warps).

• 📚 Split KV (Basic, FlashAttention-1)

// Split QKV across MMA(Warps) using naive matmul MMA&Warp tiling policy.// case: The layout of 8 MMA(2x4)  [after] kWarpTileSeqLenQxkWarpTileSeqLenK(2x2) -> 32x2,32x2=64x64:// |  [64,64]  |    warp_KV 0    |    warp_KV 1    |    warp_KV 2    |    warp_KV 3    |// | warp_QP 0 |-- MMA 0,MMA 0 --|-- MMA 2,MMA 2 --|-- MMA 4,MMA 4 --|-- MMA 6,MMA 6 --|// | warp_QP 0 |-- MMA 0,MMA 0 --|-- MMA 2,MMA 2 --|-- MMA 4,MMA 4 --|-- MMA 6,MMA 6 --|// | warp_QP 1 |-- MMA 1,MMA 1 --|-- MMA 3,MMA 2 --|-- MMA 5,MMA 5 --|-- MMA 7,MMA 7 --|// | warp_QP 1 |-- MMA 1,MMA 1 --|-- MMA 3,MMA 2 --|-- MMA 5,MMA 5 --|-- MMA 7,MMA 7 --|__global__void// Q, K, V, O -> [B, H, N, D]flash_attn_mma_stages_split_kv_kernel(half* Q, half* K, half* V, half* O, ...);

• 📚 Split Q (Faster, FlashAttention-2)

// Split Q across MMA(Warps) and keep access KV for all MMA(Warps),// in order to reduce the comm between warps via smem and warp shuffle.// case: MMA = m16n8k16, Br=16x4=64, Bc=8x8=64, layout: 4 warps// |   64x64   |      warp_KV 0       |// | warp_QP 0 | MMA 0 ... MMA 0 (x8) |// | warp_QP 1 | MMA 1 ... MMA 1 (x8) |// | warp_QP 2 | MMA 2 ... MMA 2 (x8) |// | warp_QP 3 | MMA 3 ... MMA 3 (x8) |__global__void// Q, K, V, O -> [B, H, N, D]flash_attn_mma_stages_split_q_kernel(half* Q, half* K, half* V, half* O, ...);

• 📚 Split Q + Shared KV SMEM (1/2 SRAM vs FA2)

// K, V shared the same shared memory, improve block occupancy.__global__void// Q, K, V, O -> [B, H, N, D]flash_attn_mma_stages_split_q_shared_kv_kernel(half* Q, half* K, half* V, half* O, ...);

• 📚 Split Q + Fully Shared QKV SMEM (1/4 SRAM vs FA2)

// Q, K, V fully shared the same shared memory and prefetch Q s2r, improve block occupancy// and reduce Q SMEM IO-Access.__global__void// Q, K, V, O -> [B, H, N, D]flash_attn_mma_stages_split_q_shared_qkv_kernel(half* Q, half* K, half* V, half* O, ...);

• 📚 Split Q + QK Fine-grained Tiling (O(16xd) SRAM vs FA2O(4xBrxd) SRAM,Headdim -> 1024)

// Fine-grained tiling at the MMA level for Q@K^T results in a constant SRAM usage of// 64 * kMmaAtomK for Q and K. For V, the SRAM complexity is O(kMmaAtomK * d), leading to// an overall SRAM complexity of O(kMmaAtomK * d). Consequently, this approach allows us to// extend D (head dimension) up to 1024.__global__void// Q, K, V, O -> [B, H, N, D]flash_attn_mma_stages_split_q_tiling_qk_kernel(half* Q, half* K, half* V, half* O, ...);

• 📚 Split Q + Fully QKV Fine-grained Tiling (O(2xBrx16)~O(1) SRAM vs FA2O(4xBrxd) SRAM)

// Fine-grained tiling at the MMA level for all Q@K^T and P@V results in a constant SRAM usage of// Br * 16 or Bc * 16 for Q, K, V, leading to an overall SRAM complexity of O(Br * 16). Consequently,// this approach allows us to run faster than SDPA w or w/o MMA Acc F32.__global__void// Q, K, V, O -> [B, H, N, D]flash_attn_mma_stages_split_q_tiling_qkv_kernel(half* Q, half* K, half* V, half* O, ...);

💡NOTE:📚Split Q + Fully QKV Fine-grained Tiling has been refactored into 🤖ffpa-attn.

The kernels listed here will guide you through a step-by-step progression, ranging from easy to very challenging topics. Theworkflow for each topic will be as follows: customCUDA kernel implementation -> PyTorchPython bindings -> Run tests. 👉TIPS:* = Tensor Cores (WMMA, MMA, CuTe), otherwise, CUDA Cores;/ = not supported;✔️ = supported;❔ = TODO. Contents are listed as follows:

• 📚 Easy ⭐️
• 📚 Medium ⭐️⭐️
• 📚 Hard ⭐️⭐️⭐️
• 📚 Hard+ ⭐️⭐️⭐️⭐️
• 📚 Hard++ ⭐⭐⭐️⭐️⭐️
• 📚 Triton ⭐⭐⭐️
• 📚 CUTLASS ⭐⭐⭐️

📚 Easy and📚 Medium sections cover operations such aselement-wise, mat_trans, warp/block reduce, nms, relu, gelu, swish, layer-norm, rms-norm, online-softmax, dot-prod, embedding and basic usage forFP32,FP16,BF16 andFP8 .📚 Hard,📚 Hard+ and📚 Hard++ sections delve deeper into advanced topics, primarily focusing on operations likesgemv, sgemm, hgemv, hgemm and flash-attention. These sections also provide numerous kernels implemented using Tensor Cores with pure MMA PTX.

• 📚 FlashAttention-2 MMA (MMA Acc F32/F16, swizzle, QKV smem share, fine-grained tiling, etc.🎉)

💡NOTE:rr: means reduce registers usage (ford>128);f32: means MMA accumulate with FP32 dtype, otherwise, FP16. softmax Acc dtype is always be FP32 for high precision;swizzle: now, only support smem swizzle for MMA.

• 📚 FFPA Attention MMA (1.8x~3x🎉faster vs SDPA EA, D > 256, FA2 not supported)

💡NOTE: 🤖ffpa-attn: 📚FFPA - Yet another Faster Flash Prefill Attention with O(1)🎉SRAM complexity for headdim > 256,1.8x~3x🎉faster than SDPA EA:📈L20 ~1.9x↑🎉,📈 A30 ~1.8x↑🎉,📈3080 ~2.9x↑🎉,📈4090 ~2.1x↑🎉.

💡说明: 本小节整理一些自己比较喜欢的文章。欢迎大家提PR推荐更多优秀的文章！

GNU General Public License v3.0

How to contribute? Star this repo or check🌤🌤CONTRIBUTE🎉🎉.

• flash-attention-minimal
• tiny-flash-attention
• cute-gemm
• cutlass_flash_atten_fp8
• cuda_learning
• cuda_hgemm
• cuda-tensorcore-hgemm
• How_to_optimize_in_GPU
• how-to-optim-algorithm-in-cuda
• cute_gemm
• cutlass