Repository files navigation

🤗 Diffusers is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Whether you're looking for a simple inference solution or training your own diffusion models, 🤗 Diffusers is a modular toolbox that supports both. Our library is designed with a focus onusability over performance,simple over easy, andcustomizability over abstractions.

🤗 Diffusers offers three core components:

• State-of-the-artdiffusion pipelines that can be run in inference with just a few lines of code.
• Interchangeable noiseschedulers for different diffusion speeds and output quality.
• Pretrainedmodels that can be used as building blocks, and combined with schedulers, for creating your own end-to-end diffusion systems.

We recommend installing 🤗 Diffusers in a virtual environment from PyPI or Conda. For more details about installingPyTorch andFlax, please refer to their official documentation.

Withpip (official package):

pip install --upgrade diffusers[torch]

Withconda (maintained by the community):

conda install -c conda-forge diffusers

Withpip (official package):

pip install --upgrade diffusers[flax]

Please refer to theHow to use Stable Diffusion in Apple Silicon guide.

Generating outputs is super easy with 🤗 Diffusers. To generate an image from text, use thefrom_pretrained method to load any pretrained diffusion model (browse theHub for 30,000+ checkpoints):

fromdiffusersimportDiffusionPipelineimporttorchpipeline=DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5",torch_dtype=torch.float16)pipeline.to("cuda")pipeline("An image of a squirrel in Picasso style").images[0]

You can also dig into the models and schedulers toolbox to build your own diffusion system:

fromdiffusersimportDDPMScheduler,UNet2DModelfromPILimportImageimporttorchscheduler=DDPMScheduler.from_pretrained("google/ddpm-cat-256")model=UNet2DModel.from_pretrained("google/ddpm-cat-256").to("cuda")scheduler.set_timesteps(50)sample_size=model.config.sample_sizenoise=torch.randn((1,3,sample_size,sample_size),device="cuda")input=noisefortinscheduler.timesteps:withtorch.no_grad():noisy_residual=model(input,t).sampleprev_noisy_sample=scheduler.step(noisy_residual,t,input).prev_sampleinput=prev_noisy_sampleimage= (input/2+0.5).clamp(0,1)image=image.cpu().permute(0,2,3,1).numpy()[0]image=Image.fromarray((image*255).round().astype("uint8"))image

Check out theQuickstart to launch your diffusion journey today!

We ❤️ contributions from the open-source community!If you want to contribute to this library, please check out ourContribution guide.You can look out forissues you'd like to tackle to contribute to the library.

• SeeGood first issues for general opportunities to contribute
• SeeNew model/pipeline to contribute exciting new diffusion models / diffusion pipelines
• SeeNew scheduler

Also, say 👋 in our public Discord channel. We discuss the hottest trends about diffusion models, help each other with contributions, personal projects or just hang out ☕.

• https://github.com/microsoft/TaskMatrix
• https://github.com/invoke-ai/InvokeAI
• https://github.com/InstantID/InstantID
• https://github.com/apple/ml-stable-diffusion
• https://github.com/Sanster/lama-cleaner
• https://github.com/IDEA-Research/Grounded-Segment-Anything
• https://github.com/ashawkey/stable-dreamfusion
• https://github.com/deep-floyd/IF
• https://github.com/bentoml/BentoML
• https://github.com/bmaltais/kohya_ss
• +14,000 other amazing GitHub repositories 💪

Thank you for using us ❤️.

This library concretizes previous work by many different authors and would not have been possible without their great research and implementations. We'd like to thank, in particular, the following implementations which have helped us in our development and without which the API could not have been as polished today:

• @CompVis' latent diffusion models library, availablehere
• @hojonathanho original DDPM implementation, availablehere as well as the extremely useful translation into PyTorch by @pesser, availablehere
• @ermongroup's DDIM implementation, availablehere
• @yang-song's Score-VE and Score-VP implementations, availablehere

We also want to thank @heejkoo for the very helpful overview of papers, code and resources on diffusion models, availablehere as well as @crowsonkb and @rromb for useful discussions and insights.

@misc{von-platen-etal-2022-diffusers,author ={Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Dhruv Nair and Sayak Paul and William Berman and Yiyi Xu and Steven Liu and Thomas Wolf},title ={Diffusers: State-of-the-art diffusion models},year ={2022},publisher ={GitHub},journal ={GitHub repository},howpublished ={\url{https://github.com/huggingface/diffusers}}}