Console

<a href="https://console.cloud.google.com/vertex-ai/studio/media/generate;tab=image"target="console" track-name="consoleLink" track-type="task">Go to MediaStudio</a>

1. ClickUpload and select a file to upload.

2. ClickInpaint.

   Do one of the following:

   • Upload your own mask:

     1. Create a mask on your computer.

     2. ClickUpload mask and select a mask to upload.

   • Define your mask: in the editing toolbar, use the mask tools (check_box_outline_blankbox,brushbrush, ormasked_transitionsinvert tool) to specify the area or areas to add content to.

3. Optional: In theParameters panel, adjust the following options:

   • Model: the Imagen model to use.

   • Number of results: the number of result to generate.

   • Negative prompt: describe what you want to exclude from thegenerated images.

4. In the prompt field, enter a prompt to modify the image.

5. Clickplay_arrowGenerate.

Python

Install

pip install --upgrade google-genai

To learn more, see the SDK reference documentation.

Set environment variables to use the Gen AI SDK with Vertex AI:

# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values# with appropriate values for your project.exportGOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECTexportGOOGLE_CLOUD_LOCATION=us-central1exportGOOGLE_GENAI_USE_VERTEXAI=True

fromgoogleimportgenaifromgoogle.genai.typesimport(RawReferenceImage,MaskReferenceImage,MaskReferenceConfig,EditImageConfig,)client=genai.Client()# TODO(developer): Update and un-comment below line# output_file = "output-image.png"raw_ref=RawReferenceImage(reference_image=Image.from_file(location="test_resources/fruit.png"),reference_id=0,)mask_ref=MaskReferenceImage(reference_id=1,reference_image=Image.from_file(location="test_resources/fruit_mask.png"),config=MaskReferenceConfig(mask_mode="MASK_MODE_USER_PROVIDED",mask_dilation=0.01,),)image=client.models.edit_image(model="imagen-3.0-capability-001",prompt="A plate of cookies",reference_images=[raw_ref,mask_ref],config=EditImageConfig(edit_mode="EDIT_MODE_INPAINT_INSERTION",),)image.generated_images[0].image.save(output_file)print(f"Created output image using{len(image.generated_images[0].image.image_bytes)} bytes")# Example response:# Created output image using 1234567 bytes

REST

Before using any of the request data, make the following replacements:

• REGION: The region that your project is located in. For more information about supported regions, seeGenerative AI on Vertex AI locations.
• PROJECT_ID: Your Google Cloud project ID.
• TEXT_PROMPT: Optional. A text prompt to guide the images that the model generates. For best results, use a description of the masked area and avoid single-word prompts. For example, use "a cute corgi" instead of "corgi".
• B64_BASE_IMAGE: A base64-encoded image of the image being edited that is 10MB or less in size. For more information about base64-encoding, seeBase64 encode and decode files.
• B64_MASK_IMAGE: A base64-encoded black and white mask image that is 10MB or less in size.
• MASK_DILATION: Optional. A float value between 0 and 1, inclusive, that represents the percentage of the image width to grow the mask by. Usingdilation helps compensate for imprecise masks. We recommend a value of0.01.

• EDIT_STEPS: Optional. An integer that represents the number of sampling steps. A higher value offers better image quality, a lower value offers better latency.

  We recommend that you try35 steps to start. If the quality doesn't meet your requirements, then we recomment increasing the value towards an upper limit of75.

• SAMPLE_COUNT: Optional. An integer that describes the number of images to generate. The accepted range of values is1-4. The default value is4.

HTTP method and URL:

POST https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/imagen-3.0-capability-001:predict

Request JSON body:

{  "instances": [    {      "prompt": "TEXT_PROMPT",      "referenceImages": [        {          "referenceType": "REFERENCE_TYPE_RAW",          "referenceId": 1,          "referenceImage": {            "bytesBase64Encoded": "B64_BASE_IMAGE"          }        },        {          "referenceType": "REFERENCE_TYPE_MASK",          "referenceImage": {            "bytesBase64Encoded": "B64_MASK_IMAGE"          },          "maskImageConfig": {            "maskMode": "MASK_MODE_USER_PROVIDED",            "dilation":MASK_DILATION          }        }      ]    }  ],  "parameters": {    "editConfig": {      "baseSteps":EDIT_STEPS    },    "editMode": "EDIT_MODE_INPAINT_INSERTION",    "sampleCount":SAMPLE_COUNT  }}

To send your request, choose one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/imagen-3.0-capability-001:predict"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/imagen-3.0-capability-001:predict" | Select-Object -Expand Content

{  "predictions": [    {      "bytesBase64Encoded": "BASE64_IMG_BYTES",      "mimeType": "image/png"    },    {      "mimeType": "image/png",      "bytesBase64Encoded": "BASE64_IMG_BYTES"    }  ]}

Console

<a href="https://console.cloud.google.com/vertex-ai/studio/media/generate;tab=image"target="console" track-name="consoleLink" track-type="task">Go to MediaStudio</a>

1. ClickUpload and select a file to upload.

2. ClickInpaint.

3. In the editing toolbar, clickbackground_replaceExtract mask.

4. Select one of the mask extraction options:

   • Background elements: detects the background elements and creates a mask around them.

   • wallpaperForeground elements:detects the foreground objects and creates a mask around them.

   • background_replacePeople:detects people and creates a mask around them.

5. Optional: In theParameters panel, adjust the followingoptions:

   • Model: the Imagen model to use.

   • Number of results: the number of result to generate.

   • Negative prompt: describe what you want to exclude from the generated images.

6. In the prompt field, enter a prompt to modify the image.

7. ClicksendGenerate.

Python

Install

pip install --upgrade google-genai

To learn more, see the SDK reference documentation.

Set environment variables to use the Gen AI SDK with Vertex AI:

# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values# with appropriate values for your project.exportGOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECTexportGOOGLE_CLOUD_LOCATION=us-central1exportGOOGLE_GENAI_USE_VERTEXAI=True

fromgoogleimportgenaifromgoogle.genai.typesimport(RawReferenceImage,MaskReferenceImage,MaskReferenceConfig,EditImageConfig,)client=genai.Client()# TODO(developer): Update and un-comment below line# output_file = "output-image.png"raw_ref=RawReferenceImage(reference_image=Image.from_file(location="test_resources/fruit.png"),reference_id=0,)mask_ref=MaskReferenceImage(reference_id=1,reference_image=None,config=MaskReferenceConfig(mask_mode="MASK_MODE_FOREGROUND",mask_dilation=0.1,),)image=client.models.edit_image(model="imagen-3.0-capability-001",prompt="A small white ceramic bowl with lemons and limes",reference_images=[raw_ref,mask_ref],config=EditImageConfig(edit_mode="EDIT_MODE_INPAINT_INSERTION",),)image.generated_images[0].image.save(output_file)print(f"Created output image using{len(image.generated_images[0].image.image_bytes)} bytes")# Example response:# Created output image using 1234567 bytes

REST

Before using any of the request data, make the following replacements:

• PROJECT_ID: Your Google Cloudproject ID.
• LOCATION: Your project's region. For example,us-central1,europe-west2, orasia-northeast3. For a list of available regions, seeGenerative AI on Vertex AI locations.
• TEXT_PROMPT: The text prompt guides what images the model generates. When you use a prompt for inpainting insertion, use a description of the masked area for best results. Avoid single-word prompts. For example, use "a cute corgi" instead of "corgi".
• B64_BASE_IMAGE: The base image to edit or upscale. The image must be specified as abase64-encoded byte string. Size limit: 10 MB.
• MASK_MODE - A string that sets the type of automatic mask creation the model uses. Available values:
  • MASK_MODE_BACKGROUND: Automatically generates a mask using background segmentation.
  • MASK_MODE_FOREGROUND: Automatically generates a mask using foreground segmentation.
  • MASK_MODE_SEMANTIC: Automatically generates a mask using semantic segmentation based on thesegmentation classes you specify in themaskImageConfig.maskClasses array. For example:

              "maskImageConfig": {            "maskMode": "MASK_MODE_SEMANTIC",            "maskClasses": [175, 176], // bicycle, car            "dilation": 0.01          }

• MASK_DILATION - float. The percentage of image width to dilate this mask by. A value of0.01 is recommended to compensate for imperfect input masks.
• EDIT_STEPS - integer. The number of sampling steps for the base model. For inpainting insertion, start at35 steps. Increase steps to upper limit of75 if the quality doesn't meet your requirements. Increasing steps also increases request latency.
• EDIT_IMAGE_COUNT - The number of edited images. Accepted integer values: 1-4. Default value: 4.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagen-3.0-capability-001:predict

Request JSON body:

{  "instances": [    {      "prompt": "TEXT_PROMPT",      "referenceImages": [        {          "referenceType": "REFERENCE_TYPE_RAW",          "referenceId": 1,          "referenceImage": {            "bytesBase64Encoded": "B64_BASE_IMAGE"          }        },        {          "referenceType": "REFERENCE_TYPE_MASK",          "referenceId": 2,          "maskImageConfig": {            "maskMode": "MASK_MODE",            "dilation":MASK_DILATION          }        }      ]    }  ],  "parameters": {    "editConfig": {      "baseSteps":EDIT_STEPS    },    "editMode": "EDIT_MODE_INPAINT_INSERTION",    "sampleCount":EDIT_IMAGE_COUNT  }}

To send your request, choose one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagen-3.0-capability-001:predict"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagen-3.0-capability-001:predict" | Select-Object -Expand Content

{  "predictions": [    {      "bytesBase64Encoded": "BASE64_IMG_BYTES",      "mimeType": "image/png"    },    {      "mimeType": "image/png",      "bytesBase64Encoded": "BASE64_IMG_BYTES"    }  ]}