Create and manage repositories

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

Note: To provide feedback or ask questions that are related to this Previewfeature, contactbigquery-repositories-feedback@google.com.

This document shows you how to work with repositories in BigQuery,including the following tasks:

• Creating repositories
• Deleting repositories
• Sharing repositories
• Optionally connecting a BigQuery repository to athird-party repository

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the BigQuery and Dataform APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the BigQuery and Dataform APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the APIs

Required roles

To get the permissions that you need to work with a repositories and workspaces, ask your administrator to grant you the following IAM roles on repositories and workspaces:

• Create and manage shared repositories:Code Owner (roles/dataform.codeOwner)
• Create and delete workspaces in shared repositories:Code Editor (roles/dataform.codeEditor)
• Create, modify, and version control files in workspaces in shared repositories:Code Editor (roles/dataform.codeEditor)
• View workspaces and their files in shared repositories:Code Viewer (roles/dataform.codeViewer)
• Create and manage private repositories, including all actions with workspaces and files in the private repository:Code Creator (roles/dataform.codeCreator)

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Principals that have the Code Editor role on a repository are able toedit all workspaces in the repository.

Private repositories that you create are still visible to principals who aregranted the BigQuery Admin or BigQuery Studio Admin roles at the project level.These principals can share your private repository with other users.

Create a repository

To create a BigQuery repository, follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

   If you don't see the left pane, clicklast_pageExpand left pane to open the pane.

3. In theExplorer pane, expand your project and then clickRepositories to open theRepositories tab in the details pane.

4. ClickAdd Repository.

5. In theCreate repository pane, in theRepository ID field,type a unique ID.

   IDs can only include numbers, letters, hyphens, and underscores.

6. In theRegion drop-down list, select a BigQuery regionfor storing the repository and its contents. Select the BigQueryregion nearest to your location.

   For a list of available BigQuery regions, seeBigQuery Studio locations. Therepository region does not have to match the location of yourBigQuery datasets.

7. ClickCreate.

Connect to a third-party repository

This section shows you how to connect a BigQuery repository to aremote repository. After you connect the repositories, you can perform Gitactions on the files in the workspaces contained by the repository.For example, pulling updates from the remote repository and pushing changesto the remote repository.

We recommend creating a dedicated BigQuery repository for eachremote repository that you connect to. Give the BigQueryrepository a similar name to the remote repository to help make the mappingclear.

You can connect a remote repository through HTTPS or SSH. Connecting aBigQuery repository to a remote repositorycan fail if the remote repository is not open to the public internet,for example, if it is behind a firewall. The following tablelists supported Git providers and the connection methods that are availablefor their repositories:

Git provider	Connection method
Azure DevOps Services	SSH
Bitbucket	SSH
GitHub	SSH or HTTPS
GitLab	SSH or HTTPS

Important: To create a BigQuery repository connected to a remoteGit repository that is not allow-listed in thedataform.restrictGitRemotespolicy, first add the remote Git repository to theallowedValues list in thepolicy, and then create a new BigQuery repository and connect itto the remote repository. For more information, seeRestrict remote repositories.

Connect a remote repository through SSH

To connect a remote repository through SSH, you must generate an SSH key and aSecret Manager secret. The SSH key consists of a public SSH key and aprivate SSH key. You must share the public SSH key with your Git provider,and create a Secret Manager secret with the private SSH key. Then,share the secret with your default BigQuery service account.

BigQuery uses the secret with the private SSH key to sign in toyour Git provider to commit changes on behalf of users. BigQuerymakes these commits using the user's Google Cloud email address so you can tellwho made each commit.

Warning: The private SSH key that you create is shared among allBigQuery users who use the repository. We recommend that youcreate a machine user with your Git provider and limit its access to the remoteGit repositories you plan to use with BigQuery.Only Google Cloud project owners and BigQuery users with theCode Owner role can use the key toconnect repositories. BigQuery users are not able to see thekey itself.

To connect a remote repository to aBigQuery repository through SSH, follow these steps:

1. In your Git provider, do the following:

   Azure DevOps Services

   1. In Azure DevOps Services,create a private SSH key.
   2. Upload the public SSH keyto your Azure DevOps Services repository.

   Bitbucket

   1. In Bitbucket,create a private SSH key.
   2. Upload the public SSH keyto your Bitbucket repository.

   GitHub

   1. In GitHub,check for existing SSH keys.
   2. If you don't have any existing SSH keys, or you'd like to use anew key,create a private SSH key.
   3. Upload the GitHub public SSH keyto your GitHub repository.

   GitLab

   1. In GitLab,create a private SSH key.
   2. Upload the GitLab public SSH keyto your GitLab repository.

2. In Secret Manager,create a secretand paste in your private SSH key as the secret value. Your private SSHkey should be stored in a file similar to~/.ssh/id_ed25519. Give a nameto the secret so you can find it in the future.

3. Grant access to the secret to your default Dataform service agent.

   Your default Dataform service account is in the following format:

   service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com

4. Grant theroles/secretmanager.secretAccessor roleto the service account.

5. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

6. In the left pane, clickexploreExplorer:

   

7. In theExplorer pane, expand your project and then clickRepositories to open theRepositories tab in the details pane.

8. Select the BigQuery repository that you want to connectto the remote repository.

9. In the editor, select theConfiguration tab.

10. ClickConnect with Git.

11. In theConnect to remote repository pane, select theSSH radiobutton.

12. In theRemote Git repository URL field, type the URL of the remote Gitrepository, ending with.git.

    The URL of the remote Git repository must be in one ofthe following formats:

    • Absolute URL:ssh://git@{host_name}[:{port}]/{repository_path},port is optional.
    • SCP-like URL:git@{host_name}:{repository_path}.

13. In theDefault remote branch name field, type the nameof the main branch of the remote Git repository.

14. In theSecret drop-down, select the secret that you created that containsthe SSH private key.

15. In theSSH public host key value field, type the public host key of yourGit provider.

    Azure DevOps Services

    1. To retrieve the Azure DevOps Services public host key, run the following command in the terminal:

       ssh-keyscan -t rsa ssh.dev.azure.com

    2. Copy one of the outputted keys, omittingssh.dev.azure.com from the beginning of the line.The value that you copy must be in the following format:

       ALGORITHMBASE64_KEY_VALUE

       For example:

       ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

       Verify this key is still up-to-date with Azure DevOps Services.

    Bitbucket

    1. To retrieve the Bitbucket public host key, run the following command in the terminal:

       curl https://bitbucket.org/site/ssh

    2. The command returns a list of public host keys. Choose one of the keys from the list, and copy it, omittingbitbucket.org from the beginning of the line.The value that you copy must be in the following format:

       ALGORITHMBASE64_KEY_VALUE

       For example:

       ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO

       Verify this key is still up-to-date with Bitbucket.

    GitHub

    1. To retrieve the GitHub public host key, seeGitHub's SSH key fingerprints.

    2. The page contains a list of public host keys. Choose one of them, and copy it, omittinggithub.com from the beginning of the line.The value that you copy must be in the following format:

       ALGORITHMBASE64_KEY_VALUE

       For example:

       ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

       Verify this key is still up-to-date with GitHub.

    GitLab

    1. To retrieve the GitLab public host key, seeSSHknown_hosts entries.

    2. The page contains a list of public host keys. Choose one of them, and copy it, omittinggitlab.com from the beginning of the line.The value that you copy must be in the following format:

       ALGORITHMBASE64_KEY_VALUE

       For example:

       ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf

       Verify this key is still up-to-date with GitLab.

16. ClickConnect.

Connect a remote repository through HTTPS

To connect a remote repository through HTTPS, you must create aSecret Manager secret with a personal access token, and share thesecret with your default BigQuery service account.

BigQuery then uses the access token to sign in to your Gitprovider to commit changes on behalf of users. BigQuery makesthese commits using the user's Google Cloud email address so you can tellwho made each commit.

Warning: The private HTTPS token that you create is shared among allBigQuery users who use the repository. We recommend that youcreate a machine user with your Git provider and limit its access to the remoteGit repositories you plan to use with BigQuery.Only Google Cloud project owners and BigQuery users with theCode Owner role can use the token toconnect repositories. BigQuery users are not able to see thetoken itself.

To connect a remote repository to a BigQuery repository throughHTTPS, follow these steps:

1. In your Git provider, do the following:

   GitHub

   1. In GitHub, create afine-grained personal access tokenor aclassic personal access token.

      • For a fine-grained GitHub personal access token, do the following:

      1. Select repository access to only selected repositories, then select therepository that you want to connect to.

      2. Grant read and write access on contents of the repository.

      3. Set a token expiration time appropriate to your needs.

      • For a classic GitHub personal access token, do the following:

      1. Grant BigQuery therepo permission.

      2. Set a token expiration time appropriate to your needs.

   2. If your organization uses SAML single sign-on (SSO),authorize the token.

   GitLab

   1. In GitLab, create aGitLab personal access token.

   2. Name the tokendataform; this is required.

   3. Grant BigQuery theapi,read_repository,andwrite_repository permissions.

   4. Set a token expiration time appropriate to your needs.

2. In Secret Manager,create a secretcontaining the personal access token of your remote repository.

3. Grant access to the secret to your default Dataform service agent.

   Your default Dataform service agent is in the following format:

   service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com

4. Grant theroles/secretmanager.secretAccessor roleto the service account.

5. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

6. In the left pane, clickexploreExplorer:

   

7. In theExplorer pane, expand your project and then clickRepositories to open theRepositories tab in the details pane.

8. Select the BigQuery repository that you want to connectto the remote repository.

9. In the editor, select theConfiguration tab.

10. ClickConnect with Git.

11. In theConnect to remote repository pane, select theHTTPS radiobutton.

12. In theRemote Git repository URL field, type the URL of the remote Gitrepository, ending with.git.

    The URL of the remote Git repository can't contain usernames or passwords.

13. In theDefault remote branch name field, type the nameof the main branch of the remote Git repository.

14. In theSecret drop-down, select the secret that you created that containsthe personal access token.

15. ClickConnect.

Edit the remote repository connection

To edit a connection between a BigQuery repository and a remoteGit repository, follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

3. In theExplorer pane, expand your project and then clickRepositories to open theRepositories tab in the details pane.

4. Select the BigQuery repository whose connection you want toedit.

5. In the editor, select theConfiguration tab.

6. On the repository page, clickEdit Git connection.

7. Edit connection settings.

8. ClickUpdate.

Share a repository

To share a repository, follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

3. In theExplorer pane, expand your project and then clickRepositories to open theRepositories tab in the details pane.

4. Find the repository that you want to share.

5. Clickmore_vertOpen actions and then clickShare.

6. In theShare permissions pane, clickAdd User/Group.

7. In theAdd User/Group pane, in theNew Principals field, type oneor more user or group names, separated by commas.

8. In theRole field, choose the role to assign to the new principals.

9. ClickSave.

Note: If you enhance security by setting theenable_private_workspace field(Preview)totrue in theprojects.locations.updateConfig Dataform API methodfor your Google Cloud project, then for any Dataform repository usedby your BigQuery repository, only the creator of aDataform workspace in that Dataform repository canread and write code in that Dataform workspace.

Delete a repository

To delete a repository and all its contents, follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

3. In theExplorer pane, expand your project and then clickRepositories to open theRepositories tab in the details pane.

4. Find the repository that you want to delete.

5. Clickmore_vertOpen actions and then clickDelete.

6. ClickDelete.

What's next

• Learn how tocreate workspaces.