Repository files navigation

• A demo application that uses theCLIP model for natural language media search (searching images with text, and searching related images with an image).
• Built usingPhoenix Framework,Bumblebee,Axon,Nx andHNSWLib.

• This uses Nx Servings for serving the CLIP model. There are two sets of Nx Servings in the codebase:
  1. Nx Servings provided by Bumblebee for text & image embeddings: Using ready made Nx Servings provided by Bumblebee library.
  2. Hand rolled Nx Servings for text & image embeddings: Custom implemented Nx Servings intended to learn how to implement Nx Servings from scratch.
• Both provide the same output and can be used interchangeably. However, if you're interested in learning how Nx Serving works and how to implement them, the hand rolled Nx Serving files will be helpful.

• Uses Nix for dependency management.Install Nix if you don't have it already.
• Clone the repository and rundirenv allow to activate the environment.
• To install dependencies, executerun deps.
• To start the server, executerun server.

• Create a directorypriv/images and copy all your images to this directory.
• Run the functionbuild_index to create an index from the images. It will vectorize the images, create index and save it topriv/clip_index.ann andpriv/clip_index_filenames.json files. To run the function, start the server usingrun mix phx.server and then run the function iniex shell usingMediaSearchDemo.Clip.build_index().

• Download the dataset fromhttps://unsplash.com/data/lite/latest.
• Extract and copy thephotos.tsv000 file topriv directory. (You can directly download thephotos.tsv file fromhere without downloading the whole dataset).
• Run the scriptdownload_unsplash_dataset.ex byrun elixir priv/scripts/download_unsplash_dataset.ex to download the images from the dataset. It will concurrently download images topriv/images directory.
• Once the images are downloaded topriv/images directory, you have two options:
  1. Follow the steps inUsing with Your Images section to create an index from the 25k Unsplash images. (will take some time)
  2. Download the pre-built index files fromhere andhere and save both topriv directory.

• The application uses theCLIP model with Bumblebee and Nx to create an index of images and then search the index for related images.
• For more details, please check the talk slides. Slides can be foundhere