ring-attention

Star

Here are 4 public repositories matching this topic...

feifeibear /long-context-attention

Star451

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

pytorch attention-is-all-you-need llm-training llm-inference ring-attention deepspeed-ulysses

UpdatedMar 20, 2025
Python

InternLM /InternEvo

Star368

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

pytorch multi-modal gemma pipeline-parallelism transformers-models tensor-parallelism llava llm-training internlm flash-attention zero3 llm-framework sequence-parallelism internlm2 ring-attention deepspeed-ulysses llama3 910b

UpdatedMar 20, 2025
Python

DAMO-NLP-SG /Inf-CLIP

Star235

[CVPR 2025] The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A super memory-efficiency CLIP training scheme.

memory-efficient clip contrastive-learning flash-attention ring-attention infinite-batch-size

UpdatedJan 16, 2025
Python

dtunai /Ring-Attention-Jax

Star1

Packaged Ring Attention with Blockwise Transformers for Near-Infinite Context implemented in Jax + Flax.

jax ring-attention

UpdatedMay 10, 2024
Python

Improve this page

Add a description, image, and links to thering-attention topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thering-attention topic, visit your repo's landing page and select "manage topics."

Learn more

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly