Add self-hosted GitHub runners

To add self-hosted GitHub runners, follow the instructions inadding self-hostedrunnersin the GitHub documentation.

Identify the GitHub repository

In this tutorial, theGITHUB_REPO variable represents the repositoryname. This is the part of the name that you find after the domain namefor both personal user repositories and organization repositories. For example:

• If your domain URL ishttps://github.com/myuser/myrepo, theGITHUB_REPO ismyuser/myrepo.
• If your domain URL ishttps://github.com/mycompany/ourrepo, theGITHUB_REPO ismycompany/ourrepo.

Create access token

Create a GitHub access token to dynamically add and remove runners by interactingwith your selected repository. To create an access token onGitHub and save it in the Secret Manager, follow these steps:

1. Ensure you are logged into your GitHub account.
2. Navigate to GitHub'sSettings > Developer Settings > Personal Access Tokens > Tokens (classic) page.
3. ClickGenerate new token, and selectGenerate new token (classic).
4. For the token scope, select therepo checkbox.
5. ClickGenerate token.
6. Copy the generated token.

For more information on access tokens seeAuthentication requirements inGitHub documentation.

Create a secret for your access token using Secret Manager

Take the secret token you created in theprevious step, and store it inSecret Manager. To set access permissions, follow these steps:

1. Create the secret in Secret Manager:

   echo-n"GITHUB_TOKEN"|gcloudsecretscreategithub_runner_token--data-file=-

   Replace theGITHUB_TOKEN with the value you copied from GitHub.

2. Grant theroles/secretmanager.secretAccessor to your custom service account to access your newly created secret:

   gcloudsecretsadd-iam-policy-bindinggithub_runner_token\--member"serviceAccount:$CREMA_SERVICE_ACCOUNT_NAME"\--role"roles/secretmanager.secretAccessor"

Deploy a worker pool

Create a Cloud Run worker pool to process GitHub actions. Thispool will use an image based on the GitHub-createdactions/runner image. To deploy a worker pool, follow these steps:

1. Clone the sample repository to your local machine to retrieve the code sample for use:

   gitclonehttps://github.com/GoogleCloudPlatform/cloud-run-samples

2. Change to the directory that contains the Cloud Run samplecode:

   cdcloud-run-samples/github-runner/worker-pool-container

3. Deploy the worker pool:

   gcloudbetarunworker-poolsdeployWORKER_POOL_NAME\--regionus-central1\--source.\--scaling1\--set-env-varsGITHUB_REPO=GITHUB_REPO\--set-secretsGITHUB_TOKEN=github_runner_token:latest\--service-account$CREMA_SERVICE_ACCOUNT_NAME\--memory2Gi\--cpu4

   Replace the following:

   • WORKER_POOL_NAME: the name of the worker pool
   • WORKER_POOL_LOCATION: the region of the worker pool
   • GITHUB_REPO: theGitHub repo name

   If this is the first time using Cloud Run source deploys in thisproject, Cloud Run prompts you to create a default Artifact Registry repository.

Understand the code sample

The worker pool is configured with a Dockerfile that is based on the GitHub-createdactions/runner image:

FROMghcr.io/actions/actions-runner:2.329.0# Add scripts with right permissions.USERroot# hadolint ignore=DL3045COPYstart.shstart.shRUNchmod+xstart.sh# Add start entrypoint with right permissions.USERrunnerENTRYPOINT["./start.sh"]

This helper script runs when the container is started, registering itself to theconfigured repository as an ephemeral instance, using a token you create.

# Configure the current runner instance with URL, token and name.mkdir/home/docker/actions-runner &&cd/home/docker/actions-runnerecho"GitHub Repo:${GITHUB_REPO_URL} for${RUNNER_PREFIX}-${RUNNER_SUFFIX}"./config.sh--unattended--url${GITHUB_REPO_URL}--pat${GH_TOKEN}--name${RUNNER_NAME}# Function to cleanup and remove runner from Github.cleanup(){echo"Removing runner..."./config.shremove--unattended--pat${GH_TOKEN}}# Trap signals.trap'cleanup; exit 130'INTtrap'cleanup; exit 143'TERM# Run the runner../run.sh &wait$!

Use the worker pool to accept jobs from GitHub actions

Your worker pool instance is ready to accept jobs from GitHub actions.

If your repo doesn't already have any GitHub actions, follow the instructions in thequickstart for creating your first workflow.

If your repo has GitHub actions, verify that you completed the setup of yourself-hosted runner by invoking a GitHub action on your repository.

If your GitHub action doesn't use self-hosted runners,change yourGitHub action's job fromruns-on value toself-hosted.

Once you can configured an action to use the self-hosted runners, run the action.

Confirm theaction completes successfully in the GitHub interface.

Deploy the autoscaler CREMA service

You deployed one worker in your original pool, which allows processing ofone action at a time. Depending on your Continuous Integration (CI) usage, youmight need to scale your pool to handle an influx of work to be done.

Once you deploy the worker pool with an active GitHub runner, configure theCREMA autoscaler to provision worker instances based on the job status in the actionsqueue.

This implementation listens for aworkflow_jobevent. When you create a workflow job, it scales up the worker pool, and oncethe job is completed, scales it down again. It won't scale the pool beyond themaximum number of instances you configure, and scales to zero when all runningjobs have completed.

You can adapt CREMA based on your workloads.

Configure the autoscaler

This tutorial uses theParameter Manager tostore the YAML configuration file for CREMA.

1. Create a parameter in the Parameter Manager to store parameter versionsfor CREMA:

   PARAMETER_ID=crema-configPARAMETER_REGION=globalgcloudparametermanagerparameterscreate$PARAMETER_ID--location=$PARAMETER_REGION--parameter-format=YAML

2. Create a YAML file,my-crema-config.yaml in the parent directory to define the autoscaler configuration:

   apiVersion:crema/v1kind:CremaConfigmetadata:name:gh-demospec:pollingInterval:10triggerAuthentications:-metadata:name:github-trigger-authspec:gcpSecretManager:secrets:-parameter:personalAccessTokenid:github_runner_tokenversion:latestscaledObjects:-spec:scaleTargetRef:name:projects/PROJECT_ID/locations/us-central1/workerpools/WORKER_POOL_NAMEtriggers:-type:github-runnername:GITHUB_RUNNERmetadata:owner:REPOSITORY_OWNERrunnerScope:reporepos:REPOSITORY_NAMEtargetWorkflowQueueLength:1authenticationRef:name:github-trigger-authadvanced:horizontalPodAutoscalerConfig:behavior:scaleDown:stabilizationWindowSeconds:10policies:-type:Podsvalue:100periodSeconds:10scaleUp:stabilizationWindowSeconds:10policies:-type:Podsvalue:2periodSeconds:10

   Replace the following:

   • PROJECT_ID: the Google Cloud project ID
   • WORKER_POOL_NAME: the name of the worker pool you deployed
   • GITHUB_RUNNER: the name of the GitHub runner you configured
   • REPOSITORY_OWNER: the owner of the GitHub repository
   • REPOSITORY_NAME: the GitHub repository name

3. Upload your local YAML file as a new parameter version:

   LOCAL_YAML_CONFIG_FILE=my-crema-config.yamlPARAMETER_VERSION=1gcloudparametermanagerparametersversionscreate$PARAMETER_VERSION\--location=$PARAMETER_REGION\--parameter=$PARAMETER_ID\--payload-data-from-file=$LOCAL_YAML_CONFIG_FILE

Grant additional permissions to your custom service account

To scale the worker pool you specified in your YAML configuration, grant the following permissionson the custom service account:

1. Grant your CREMA service account permission to read from the Parameter Manager:

   gcloudprojectsadd-iam-policy-binding$PROJECT_ID\--member="serviceAccount:$CREMA_SERVICE_ACCOUNT_NAME"\--role="roles/parametermanager.parameterViewer"

2. Grant your CREMA service account theroles/run.developer role on the worker pool:

   WORKER_POOL_NAME=WORKER_POOL_NAMEWORKER_POOL_REGION=us-central1gcloudbetarunworker-poolsadd-iam-policy-binding$WORKER_POOL_NAME\--region=$WORKER_POOL_REGION\--member="serviceAccount:$CREMA_SERVICE_ACCOUNT_NAME"\--role="roles/run.developer"

   ReplaceWORKER_POOL_NAME with the name of the worker pool.

3. Grant your CREMA service account permission to write metrics:

   gcloudprojectsadd-iam-policy-binding$PROJECT_ID\--member="serviceAccount:$CREMA_SERVICE_ACCOUNT_NAME"\--role="roles/monitoring.metricWriter"

4. Grant your CREMA service account the service account user role:

   gcloudprojectsadd-iam-policy-binding$PROJECT_ID\--member="serviceAccount:$CREMA_SERVICE_ACCOUNT_NAME"\--role="roles/iam.serviceAccountUser"

Deploy the service to scale your workloads

To deploy the service to scale your worker pool, run the following command witha prebuilt container image:

SERVICE_NAME=my-crema-serviceSERVICE_REGION=us-central1CREMA_CONFIG_PARAM_VERSION=projects/$PROJECT_ID/locations/$PARAMETER_REGION/parameters/$PARAMETER_ID/versions/$PARAMETER_VERSIONIMAGE=us-central1-docker.pkg.dev/cloud-run-oss-images/crema-v1/autoscaler:1.0gcloudbetarundeploy$SERVICE_NAME\--image=${IMAGE}\--region=${SERVICE_REGION}\--service-account="${CREMA_SERVICE_ACCOUNT_NAME}"\--no-allow-unauthenticated\--no-cpu-throttling\--base-image=us-central1-docker.pkg.dev/serverless-runtimes/google-22/runtimes/java21\--labels=created-by=crema\--set-env-vars="CREMA_CONFIG=${CREMA_CONFIG_PARAM_VERSION},OUTPUT_SCALER_METRICS=True"

Create webhook secret value

To create a secret value to access the GitHub webhook, do the following:

1. Create a Secret Manager secret to manage access to your GitHub webhook.

   echo-n"WEBHOOK_SECRET"|gcloudsecretscreategithub_webhook_secret--data-file=-

   ReplaceWEBHOOK_SECRET with an arbitrary string value.

2. Grant access to the secret to the autoscaler service account:

   gcloudsecretsadd-iam-policy-bindinggithub_webhook_secret\--member"serviceAccount:$CREMA_SERVICE_ACCOUNT_NAME"\--role"roles/secretmanager.secretAccessor"

Create GitHub webhook

To create the GitHub webhook, follow these steps:

1. Ensure you are logged into your GitHub account.
2. Navigate to your GitHub repository.
3. ClickSettings.
4. UnderCode and automation, clickWebhooks.
5. ClickAdd webhook.

6. Enter the following:

   1. InPayload URL, enter the URL of the Cloud Run CREMA service youdeployed,my-crema-service.
   2. ForContent type, selectapplication/json.
   3. ForSecret, enter theWEBHOOK_SECRET value you created previously.
   4. ForSSL verification, selectEnable SSL verification.
   5. ForWhich events would you like to trigger this webhook?, selectLet me select individual events.
   6. In the event selection, selectWorkflow jobs. Unselect any other option.
   7. ClickAdd webhook.

Test your CREMA service

To verify your autoscaling service is working correctly,checktheLogs tab of the Cloud Run service.

You should see the followinglogs in your service's logs each time metrics are refreshed:

Each log message is labeled with the component that emitted it.

[INFO] [METRIC-PROVIDER] Starting metric collection cycle[INFO] [METRIC-PROVIDER] Successfully fetched scaled object metrics ...[INFO] [METRIC-PROVIDER] Sending scale request ...[INFO] [SCALER] Received ScaleRequest ...[INFO] [SCALER] Current instances ...[INFO] [SCALER] Recommended instances ...