Repository files navigation

CLIP, or Contrastive Language-Image Pretraining, is a cutting-edge AI model that comprehends and connects text and images, revolutionizing how we interpret online data.

This library provides you with an instant, easy-to-use interface for CLIP, allowing you to harness its capabilities without any setup hassles. BentoML takes care of all the complexity of serving the model!

Ensure that you have Python 3.8 or newer andpip installed on your system. We highly recommend using a Virtual Environment to avoid any potential package conflicts.

To install the service, enter the following command:

pip install clip-api-service

Once the installation process is complete, you can start the service by running:

clip-api-service serve --model-name=ViT-B-32:openai

Your service is now running! Interact with it via the Swagger UI atlocalhost:3000

Or try this tutorial in Google Colab:CLIP demo.

Harness the capabilities of the CLIP API service across a range of applications:

1. Text and Image Embedding
   • Useencode to transform text or images into meaningful embeddings. This makes it possible to perform tasks such as:
     1. Neural Search: Utilize encoded embeddings to power a search engine capable of understanding and indexing images based on their textual descriptions, and vice versa.
     2. Custom Ranking: Design a ranking system based on embeddings, providing unique ways to sort and categorize data according to your context.

2. Zero-Shot Image Classification

   • Userank to perform image classification without any training. For example:
     1. Given a set of images, classify an image as being "a picture of a dog" or "a picture of a cat".
     2. More complex classifications such as recognizing different breeds of dogs can also be performed, illustrating the versatility of the CLIP API service.

3. Visual Reasoning

   • Therank function can also be used to provide reasoning about visual scenarios. For instance:

Visual Scenario	Query Image	Candidates	Output
Counting Objects		This is a picture of 1 dog This is a picture of 2 dogs This is a picture of 3 dogs	Image matched with "3 dogs"
Identifying Colors		The car is red The car is blue The car is green	Image matched with "blue car"
Understanding Motion		The car is parked The car is moving The car is turning	Image matched with "parked car"
Recognizing Location		The car is in the suburb The car is on the highway The car is in the street	Image matched with "car in the street"
Relative Positioning		The big car is on the left, the small car is on the right The small car is on the left, the big car is on the right	Image matched with the provided description

Effortlessly transition your project into a production-ready application usingBentoCloud, the production-ready platform for managing and deploying machine learning models.

Start by creating a BentoCloud account. Once you've signed up, log in to your BentoCloud account using the command:

bentoml cloud login --api-token<your-api-token> --endpoint<bento-cloud-endpoint>

Note: Replace<your-api-token> and<bento-cloud-endpoint> with your specific API token and the BentoCloud endpoint respectively.

Next, build your BentoML service using thebuild command:

clip-api-service build --model-name=ViT-B-32:openai

Then, push your freshly-built Bento service to BentoCloud using thepush command:

bentoml push<name:version>

Lastly, deploy this application to BentoCloud with a singlebentoml deployment create command following thedeployment instructions.

BentoML offers a number of options for deploying and hosting online ML services into production, learn more atDeploying a Bento.

Accepts either:

• img_uri : An Image URI, i.ehttps://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg
• text : A string
• img_blob : Base64 encoded string

Returns a vector of embeddings of length 768.

Example:

curl -X 'POST' \  'http://localhost:3000/encode' \  -H 'accept: application/json' \  -H 'Content-Type: application/json' \  -d '[  {    "img_uri": "https://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg"  },  {    "text": "picture of a dog"  },  {    "img_blob": "iVBORw0KGgoAAAANSUhEUgAAABIAAAAPCAYAAADphp8SAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AAAApdEVYdENyZWF0aW9uIFRpbWUARnJpIDI2IE1heSAyMDIzIDA0OjE2OjIxIEFNCXIaIQAAAnhJREFUOI1Fk82OY0UMhT+7qu6dJPfmZ3p6EhYR3QSGnk0P6QV/4rlYsOIBeKrhIQAJtZCgNQsmEWiQSG5V2SwSeixZtiydYx9blpsJPpv2fPP1V3zx5ee8vHnJbDFlNl/Q9GMQBwLUAM7JcXADN9wN3Im1wOaja7afbfnk4xcsVytS25DaFsyptSDiaIyAnMn8kVTOMc6mI+62d2w2G5bLFZOuBxUkJsyNYoIGRVXfTyMCLmByIseJm+trbl58ynL5nK7v0dTgOGgENyQqsWlxdxxH4FEW7og7mBHvtq+4vHxG381IqQF3qjvBnOrgKBIiOR/fA89gMUPdoRpxe/uKbjym6zton4AZ6oZoRNxQUQiJOhzO3R1xeyTiXItXV1csLi4gRjADOIHdEEBEoGSCCOaOmUGtKI4CQQMIxL7vmIwmcBygAdoWAKuV6o5GwUwQB/HzjkTOufC/xZRaSi6IOjHFU1UVrRkrmTwcERViiohDQFANiAmYYbUgBnE8miAiDDlT/j0gQ0aj4qoklZMkHBsKIShBFJXTJLVWai64GbGUQnrSEori7ljO5Gy4gMZIahMpRA7DABZAwXGsVnIulOGIuxFfv/6RUdtwcfmM2XTKfDEnNYlcCjkP1BJIqYUChuNU3J1aM+Us382Jv97f8/d+x3w+YzbtWa5WLD94ztOnCyZdR0QRM8TkdACr1FqoJeNuuFUAwg/fffu9m7Hf73j4/Q9+/uUn3jw8YG6MRyNijFgtJG0wM8owMBwPlJKppZ6+RiA2TWK9XvPhes27f96x3+04Hg/s/vyLe/2N29ueyWxCzQe8Gvu3b0GUXCu7/Y5ijgblP3zyX4rqQyp1AAAAAElFTkSuQmCC"  }]'

Accepts a list ofqueries and a list ofcandidates. Similar to above,queries andcandidates are either:

• img_uri : An Image URI, i.ehttps://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg
• text : A string
• img_blob : Base64 encoded string

Returns a list of probabilies and cosine similarities of each candidate with respect to the query.

Example:

curl -X 'POST' \  'http://localhost:3000/rank' \  -H 'accept: application/json' \  -H 'Content-Type: application/json' \  -d '{  "queries": [    {      "img_uri": "https://hips.hearstapps.com/hmg-prod/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg"    }  ],  "candidates": [    {      "text": "picture of a dog"    },    {      "text": "picture of a cat"    },    {      "text": "picture of a bird"    },    {      "text": "picture of a car"    },    {      "text": "picture of a plane"    },    {      "text": "picture of a boat"    }  ]}'

And the response looks like:

{  "probabilities": [    [      0.9958375692367554,      0.0022114247549325228,      0.001514736912213266,      0.00011969256593147293,      0.00019143625104334205,      0.0001251235808013007    ]  ],  "cosine_similarities": [    [      0.2297772467136383,      0.16867777705192566,      0.16489382088184357,      0.13951312005519867,      0.14420939981937408,      0.13995687663555145    ]  ]}

Spins up a HTTP Server with the model of your choice.

Arguments:

• --model-name : Name of the CLIP model. Uselist_models to see the list of available model. Default:openai/clip-vit-large-patch14

Builds a Bento with the model of your choice

Arguments:

• --model-name : Name of the CLIP model. Uselist_models to see the list of available model. Default:openai/clip-vit-large-patch14

List all available CLIP models.