Repository files navigation

Stable		Nightly		Discord	Build

Installation |Getting Started |Examples |High-level APIs |Low-level APIs |Cite our work

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPUtraining throughput by 20% and reducesmemory usage by 60%. We have implementedHugging Face CompatibleRMSNorm,RoPE,SwiGLU,CrossEntropy,FusedLinearCrossEntropy, and more to come. The kernel works out of the box withFlash Attention,PyTorch FSDP, andMicrosoft DeepSpeed. We welcome contributions from the community to gather the best kernels for LLM training.

We've also added optimized Post-Training kernels that deliverup to 80% memory savings for alignment and distillation tasks. We support losses like DPO, CPO, ORPO, SimPO, KTO, JSD, and many more. Check outhow we optimize the memory.

With one line of code, Liger Kernel can increase throughput by more than 20% and reduce memory usage by 60%, thereby enabling longer context lengths, larger batch sizes, and massive vocabularies.

Speed Up	Memory Reduction

Note:

• Benchmark conditions: LLaMA 3-8B, Batch Size = 8, Data Type =bf16, Optimizer = AdamW, Gradient Checkpointing = True, Distributed Strategy = FSDP1 on 8 A100s.
• Hugging Face models start to OOM at a 4K context length, whereas Hugging Face + Liger Kernel scales up to 16K.

We provide optimized post training kernels like DPO, ORPO, SimPO, and more which can reduce memory usage by up to 80%. You can easily use them as python modules.

fromliger_kernel.chunked_lossimportLigerFusedLinearORPOLossorpo_loss=LigerFusedLinearORPOLoss()y=orpo_loss(lm_head.weight,x,target)

• Ease of use: Simply patch your Hugging Face model with one line of code, or compose your own model using our Liger Kernel modules.
• Time and memory efficient: In the same spirit as Flash-Attn, but for layers likeRMSNorm,RoPE,SwiGLU, andCrossEntropy! Increases multi-GPU training throughput by 20% and reduces memory usage by 60% withkernel fusion,in-place replacement, andchunking techniques.
• Exact: Computation is exact—no approximations! Both forward and backward passes are implemented with rigorous unit tests and undergo convergence testing against training runs without Liger Kernel to ensure accuracy.
• Lightweight: Liger Kernel has minimal dependencies, requiring only Torch and Triton—no extra libraries needed! Say goodbye to dependency headaches!
• Multi-GPU supported: Compatible with multi-GPU setups (PyTorch FSDP, DeepSpeed, DDP, etc.).
• Trainer Framework Integration:Axolotl,LLaMa-Factory,SFTTrainer,Hugging Face Trainer,SWIFT,oumi

• torch >= 2.1.2
• triton >= 2.3.0

• torch >= 2.5.0 Install according to the instruction in Pytorch official webpage.
• triton >= 3.0.0 Install from pypi. (e.g.pip install triton==3.0.0)

# Need to pass the url when installingpip install -e .[dev] --extra-index-url https://download.pytorch.org/whl/nightly/rocm6.2

• transformers >= 4.x: Required if you plan to use the transformers models patching APIs. The specific model you are working will dictate the minimum version of transformers.

Note:Our kernels inherit the full spectrum of hardware compatibility offered byTriton.

To install the stable version:

$ pip install liger-kernel

To install the nightly version:

$ pip install liger-kernel-nightly

To install from source:

git clone https://github.com/linkedin/Liger-Kernel.gitcd Liger-Kernel# Install Default Dependencies# Setup.py will detect whether you are using AMD or NVIDIApip install -e.# Setup Development Dependenciespip install -e".[dev]"

There are a couple of ways to apply Liger kernels, depending on the level of customization required.

Using theAutoLigerKernelForCausalLM is the simplest approach, as you don't have to import a model-specific patching API. If the model type is supported, the modeling code will be automatically patched using the default settings.

fromliger_kernel.transformersimportAutoLigerKernelForCausalLM# This AutoModel wrapper class automatically monkey-patches the# model with the optimized Liger kernels if the model is supported.model=AutoLigerKernelForCausalLM.from_pretrained("path/to/some/model")

Using thepatching APIs, you can swap Hugging Face models with optimized Liger Kernels.

importtransformersfromliger_kernel.transformersimportapply_liger_kernel_to_llama# 1a. Adding this line automatically monkey-patches the model with the optimized Liger kernelsapply_liger_kernel_to_llama()# 1b. You could alternatively specify exactly which kernels are appliedapply_liger_kernel_to_llama(rope=True,swiglu=True,cross_entropy=True,fused_linear_cross_entropy=False,rms_norm=False)# 2. Instantiate patched modelmodel=transformers.AutoModelForCausalLM("path/to/llama/model")

You can take individualkernels to compose your models.

fromliger_kernel.transformersimportLigerFusedLinearCrossEntropyLossimporttorch.nnasnnimporttorchmodel=nn.Linear(128,256).cuda()# fuses linear + cross entropy layers together and performs chunk-by-chunk computation to reduce memoryloss_fn=LigerFusedLinearCrossEntropyLoss()input=torch.randn(4,128,requires_grad=True,device="cuda")target=torch.randint(256, (4, ),device="cuda")loss=loss_fn(model.weight,input,target)loss.backward()

• Fused Linear kernels combine linear layers with losses, reducing memory usage by up to 80% - ideal for HBM-constrained workloads.
• Other kernels use fusion and in-place techniques for memory and performance optimization.

• Contributing Guidelines
• Acknowledgements
• License Information

• Glows.ai: Sponsoring NVIDIA GPUs for our open source developers.
• AMD: Providing AMD GPUs for our AMD CI.
• Intel: Providing Intel GPUs for our Intel CI.
• Modal: Free 3000 credits from GPU MODE IRL for our NVIDIA CI.
• EmbeddedLLM: Making Liger Kernel run fast and stable on AMD.
• HuggingFace: Integrating Liger Kernel into Hugging Face Transformers and TRL.
• Lightning AI: Integrating Liger Kernel into Lightning Thunder.
• Axolotl: Integrating Liger Kernel into Axolotl.
• Llama-Factory: Integrating Liger Kernel into Llama-Factory.

• For issues, create a Github ticket in this repository
• For open discussion, joinour discord channel on GPUMode
• For formal collaboration, send an email toyannchen@linkedin.com andhning@linkedin.com

Biblatex entry:

@article{hsu2024ligerkernelefficienttriton,title={Liger Kernel: Efficient Triton Kernels for LLM Training},author={Pin-Lun Hsu and Yun Dai and Vignesh Kothapalli and Qingquan Song and Shao Tang and Siyu Zhu and Steven Shimizu and Shivam Sahni and Haowen Ning and Yanning Chen},year={2024},eprint={2410.10989},archivePrefix={arXiv},primaryClass={cs.LG},url={https://arxiv.org/abs/2410.10989},journal={arXiv preprint arXiv:2410.10989},}

↑ Back to Top ↑