This document describes a threat finding type in Security Command Center. Threat findings are generated bythreat detectors when they detecta potential threat in your cloud resources. For a full list of available threat findings, seeThreat findings index.

Overview

A binary that was not part of the original container image was executed.Attackers commonly install exploitation tooling and malware after the initialcompromise. Ensuring that your containers are immutable is an important bestpractice. This is a low-severity finding, because your organization might not befollowing this best practice. There are correspondingExecution: AddedMalicious Binary Executed findings when the hash of the binary is a knownindicator of compromise (IoC).

Container Threat Detection is the sourceof this finding.

How to respond

To respond to this finding, do the following:

Step 1: Review finding details

1. Open anAdded Binary Executed finding as directed inReviewing findings. The details panel for thefinding opens to theSummary tab.

2. On theSummary tab, review the information in the following sections:

   • What was detected, especially the following fields:
     • Program binary: the absolute path of the added binary.
     • Arguments: the arguments provided when invoking the added binary.
   • Affected resource, especially the following fields:
     • Resource full name: thefull resource nameof the cluster including the project number, location, and cluster name.
   • Related links, especially the following fields:
     • VirusTotal indicator: link to the VirusTotal analysis page.

3. Click theJSON and note the following fields:

   • resource:
     • project_display_name: the name of the project that containsthe cluster.
   • sourceProperties:
     • Pod_Namespace: the name of the Pod's Kubernetes namespace.
     • Pod_Name: the name of the GKE Pod.
     • Container_Name: the name of the affected container.
     • Container_Image_Uri: the name of the container image being deployed.
     • VM_Instance_Name: the name of the GKE node where thePod executed.

4. Identify other findings that occurred at a similar time for this container. Related findings might indicate that this activity was malicious, instead of a failure to follow best practices.

Step 2: Review cluster and node

1. In the Google Cloud console, go to theKubernetes clusters page.

   Go to Kubernetes clusters

2. On the Google Cloud console toolbar, select the project listed inresource.project_display_name, if necessary.

3. Select the cluster listed on theResource full name row in theSummary tab of the finding details. Note any metadata about the cluster and its owner.

4. Click theNodes tab. Select the node listed inVM_Instance_Name.

5. Click theDetails tab and note thecontainer.googleapis.com/instance_id annotation.

Step 3: Review Pod

1. In the Google Cloud console, go to theKubernetes Workloads page.

   Go to Kubernetes Workloads

2. On the Google Cloud console toolbar, select the project listed inresource.project_display_name, if necessary.

3. Filter on the cluster listed on theResource full name row in theSummary tab of the finding details and the Pod namespace listed inPod_Namespace, if necessary.

4. Select the Pod listed inPod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

1. In the Google Cloud console, go toLogs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project listed inresource.project_display_name, if necessary.

3. SetSelect time range to the period of interest.

4. On the page that loads, do the following:

   1. Find Pod logs forPod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
   2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
   3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate thecontainer environment directly.

1. Go to the Google Cloud console.

   Open Google Cloud console

2. On the Google Cloud console toolbar, select the project listed inresource.project_display_name, if necessary.

3. ClickActivate Cloud Shell

4. Obtain GKE credentials for your cluster by running the following commands.

   For zonal clusters:

   gcloudcontainerclustersget-credentialscluster_name--zonelocation--projectproject_name

   For regional clusters:

   gcloudcontainerclustersget-credentialscluster_name--regionlocation--projectproject_name

   Replace the following:

   • cluster_name: the cluster listed inresource.labels.cluster_name
   • location: the location listed inresource.labels.location
   • project_name: the project name listed inresource.project_display_name

5. Retrieve the added binary by running:

   kubectlcpPod_Namespace/Pod_Name:Process_Binary_Fullpath-cContainer_Namelocal_file

   Replacelocal_file with a local file path to store the added binary.

6. Connect to the container environment by running:

   kubectlexec--namespace=Pod_Namespace-tiPod_Name-cContainer_Name--/bin/sh

   This command requires the container to have a shell installed at/bin/sh.

Step 6: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type:Ingress Tool Transfer,Native API.
2. Check the SHA-256 hash value for the binary flagged as malicious onVirusTotal by clicking the link inVirusTotal indicator. VirusTotal is an Alphabet-owned service thatprovides context on potentially malicious files, URLs, domains, and IPaddresses.
3. To develop a response plan, combine yourinvestigation results with the MITRE research and VirusTotal analysis.

Step 7: Implement your response

The following response plan might be appropriate for this finding, but might also impact operations.Carefully evaluate the information you gather in your investigation to determine the best way toresolve findings.

• If the binary was intended to be included in the container, rebuild thecontainer image with the binary included. This way, the container can beimmutable.
• Otherwise, contact the owner of the project with the compromised container.
• Stop ordelete thecompromised container and replace it with anew container.

What's next

• Learnhow to work with threatfindings in Security Command Center.
• Refer to theThreat findings index.
• Learn how toreview afinding through the Google Cloud console.
• Learn about theservices thatgenerate threat findings.