Repository files navigation

A powerful open-source tool for analyzing image and video datasets founded by the authors ofXGBoost,Apache TVM &Turi Create -Danny Bickson,Carlos Guestrin andAmir Alush.

pip install fastdup fromPyPI:

pip install fastdup

More installation options are availablehere.

Initialize and run fastdup:

importfastdupfd=fastdup.create(input_dir="IMAGE_FOLDER/")fd.run()

Explore the results in a interactive web UI:

fd.explore()

Alternatively, visualize the result in a static gallery:

fd.vis.duplicates_gallery()# gallery of duplicatesfd.vis.outliers_gallery()# gallery of outliersfd.vis.component_gallery()# gallery of connected componentsfd.vis.stats_gallery()# gallery of image statistics (e.g. blur, brightness, etc.)fd.vis.similarity_gallery()# gallery of similar images

fastdup handles labeled/unlabeled datasets in image or video format, providing a range of features:

What sets fastdup apart from other similar tools:

• Quality: High-quality analysis to identify duplicates/near-duplicates, outliers, mislabels, broken images, and low-quality images.
• Scale: Highly scalable, capable of processing 400M images on a single CPU machine. Scales up to billions of images.
• Speed: Optimized C++ engine enables high performance even on low-resource CPU machines.
• Privacy: Runs locally or on your cloud infrastructure. Your data stays where it is.
• Ease of use: Works on labeled or unlabeled datasets in image or video format with support for major operating systems like MacOS, Linux and Windows.

Learn the basics of fastdup through interactive examples. View the notebooks on GitHub or nbviewer. Even better, run them on Google Colab or Kaggle, for free.

	⚡ Quickstart: Learn how to install fastdup, load a dataset and analyze it for potential issues such as duplicates/near-duplicates, broken images, outliers, dark/bright/blurry images, and view visually similar image clusters. If you're new, start here! 📌 Dataset:Oxford-IIIT Pet.
	🧹 Finding and Removing Duplicates: Learn how to how to analyze an image dataset for duplicates and near-duplicates. 📌 Dataset:Oxford-IIIT Pet.
	🖼 Finding and Removing Mislabels: Learn how to analyze an image dataset for potential image mislabels and export the list of mislabeled images for further inspection. 📌 Dataset:Food-101.
	🎁 Image Similarity Search: Perform image search in a large dataset of images. 📌 Dataset:Shopee Product Matching.
	🤗 Hugging Face Datasets: Load and analyze datasets fromHugging Face Datasets. Perfect if you already have a dataset hosted on Hugging Face hub.
	🧠 TIMM Embeddings: Compute dataset embeddings usingTIMM (PyTorch Image Models) and run fastdup over the them to surface dataset issues. Runs on CPU and GPU.
	🦖 ONNX Embeddings: Bring your own ONNX model. In this example we extract feature vectors of your images usingDINOv2 model. Runs on CPU.

See moreexamples.

Get help from the fastdup team or community members via the following channels:

Community-contributed blog posts on fastdup:

	Deploying AWS Lambda functions with Docker Container by using Custom Base Image 🖋️atahan bulus    •    🗓 16 September 2023
	Renumics: Cleaning Image Classification Datasets With fastdup and Renumics Spotlight 🖋️Daniel Klitzke    •    🗓 4 September 2023
	Roboflow: How to Reduce Dataset Size Without Losing Accuracy 🖋️Arty Ariuntuya    •    🗓 9 August 2023
	The weighty significance of data cleanliness — or as I like to call it, “cleanliness is next to model-ness” — cannot be overstated. 🖋️Alexander Lan    •    🗓 9 March 2023
	Clean Up Your Digital Life: How I Found 1929 Fully Identical Images, Dark, Bright and Blurry Shots in Minutes, For Free. 🖋️Dickson Neoh    •    🗓 23 February 2023
	fastdup: A Powerful Tool to Manage, Clean & Curate Visual Data at Scale on Your CPU - For Free. 🖋️Dickson Neoh    •    🗓 3 January 2023
	Master Data Integrity to Clean Your Computer Vision Datasets. 🖋️Paul lusztin    •    🗓 19 December 2022

What our users say:

Visual Layer offers commercial services for managing, cleaning, and curating visual data at scale.

Sign-up for free.

Not convinced? Interact with Visual Layer Cloudpublic dataset with no sign-up required.

fastdup is licensed underCreative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License.

For any more information or inquiries regarding the license, please contact us atinfo@visual-layer.com or see theLICENSE file.