Troubleshoot transfer configurations

This document is intended to help you troubleshoot the most common issuesencountered when setting up a BigQuery Data Transfer Service transfer. This documentdoes not encompass all possible error messages or issues.

If you are experiencing issues that are not covered in this document,you canrequest support.

Before contacting Cloud Customer Care, capture transfer configuration and transfer run details. For information on how to get these details, seeGet transfer details andView transfer run details and log messages.

Examine errors

If your initial transfer run fails, you can examine the details in therun history.Errors listed in the run history can help you identify an appropriate resolutionusing this document.

You can also view error messages for a specific transfer job using theLogs Explorer.The following Logs Explorer filter returns information about aspecific transfer configuration job, along with any error messages:

resource.type="bigquery_dts_config"labels.run_id="RUN_ID"resource.labels.config_id="CONFIG_ID"

Replace the following:

• RUN_ID: the ID number of a specific job run
• CONFIG_ID: the ID number of a transfer configurationjob

Before contacting Customer Care, capture any relevant information fromthe run history or Logs Explorer including any error messages.

If you useevent-driven transfers, theevent-driven transfer configuration might fail to trigger a transfer run.You can view error messages at the top of therun historypage orconfigurationpage.

General issues

When diagnosing general transfer issues, verify the following:

• Verify that you have completed all the steps in the "Before You Begin" sectionof the documentation page for your transfer type.
• The transfer configuration properties are correct.
• The user account used to create the transfer has access to the underlyingresources.

If your transfer configuration is correct, and the appropriate permissions aregranted, refer to the following for solutions to commonly encountered issues.

Error:An unexpected issue was encountered. If this issue persists, please contact customer support.

Resolution: This error typically indicates a temporary outage or an issuewithin BigQuery. Wait approximately 2 hours for the issue to beresolved. If the problem persists,request support.

Error:INTERNAL: An internal error occurred and the request could not be completed. This is usually caused by a transient issue...

Resolution: This error typically indicates a transient internal issue. Ifyou encounter this error, you can wait to see if it resolves in the nextscheduled run, or you canmanually trigger a backfillfor the affected dates. If the issue persists,request support.

Error:Quota Exceeded.

Resolution: Transfers are subject to BigQueryquotas on load jobs. If you need to increaseyour quota, contact your Google Cloud sales representative. For moreinformation, seeQuotas and limits.

If you are loading Cloud Billing exports to BigQuery, youcan encounter theQuota Exceeded error. Both the Cloud Billing exporttables, and the destination BigQuery tables created by theBigQuery Data Transfer Service service are partitioned. Choosing theoverwrite option while setting such BigQuery Data Transfer Service jobs causesthe quota errors depending on how much data is exported. Forinformation about troubleshooting quotas, seeTroubleshoot quota and limiterrors.

If the error is because of BigQuery Data Transfer Service jobs for Cloud Billingexports, then note, that since the individual CloudBilling Export tables are partitioned, so is the target table created bythe BigQuery Data Transfer Service, and hence choosing theoverwrite optionwhile setting up such data transfer jobs will result into (DML) Quota errorsdepending on how old the Billing Accounts are. For information abouttroubleshooting quotas, seeTroubleshoot quota and limit errors.

Error:The caller does not have permission.

Resolution: Confirm the signed-in account in Google Cloud console is the same as the account you select for BigQuery Data Transfer Service when creating the transfer.

• Signed-in account in Google Cloud console:

  

• Choose an account to continue to BigQuery Data Transfer Service:

  

Error:Access Denied: ... Permission bigquery.tables.get denied on table ...

Resolution: Confirm that the BigQuery Data Transfer Serviceservice agent is grantedthebigquery.dataEditor roleon the target dataset. This grant is automatically applied when creating andupdating the transfer, but it's possible that the access policy was modifiedmanually afterwards. To regrant the permission, seeGrant access to a dataset.

Error:region violates constraint constraints/gcp.resourceLocations on the resourceprojects/project_id

Resolution: This error occurs when a user tries to create a transferconfiguration in a restricted location, as specified in thelocation restriction organization policy.You can resolve this issue bychanging the organization policyto allow for the region, or by changing the transfer configuration to adestination dataset located in a region unrestricted by the organization policy.

Error:Please look into the errors[] collection for more details.

Resolution: This error can occur when a data transfer fails. For moreinformation about why the data transfer failed, you canuse Cloud Logging toview your logs. You canfind logs for aspecific run by searchingusing the transferrun_id.

Error:Network Attachment with connected endpoints cannot be deleted.

Resolution: This error can occur when a user tries to delete their networkattachments soon after they havedeleted theirtransfer. Thishappens because it can take several days after a transfer deletion before theBigQuery Data Transfer Service can fully remove all resources associated with thetransfer, which can prevent the network attachments from being deleted. Toresolve this error, wait several days before trying to delete the networkattachments. If you want to have the network attachments deleted sooner, youcancontact support.

Authorization and permission issues

The following are some common permission errors that you can encounter when youtransfer data from different data sources:

Error:BigQuery Data Transfer Service is not enabled for <project_id>

Error:BigQuery Data Transfer Service has not been used in project <project_id> before or it is disabled ...

Resolution:Verify that the service agent role is granted with the following steps:

1. In the Google Cloud console, go to theIAM & Admin page.

   Go to IAM & Admin

2. Select theInclude Google-provided role grants checkbox.

3. Verify that the service account with the nameservice-<project_number>@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.comis shown or that it has been granted the BigQuery Data Transfer Service theBigQuery Data Transfer Service Agent role.

   

If the service account is not shown, or it does not have the BigQuery Data Transfer Serviceservice agent role granted, grant the predefined role in the Google Cloud consoleor by running the following Google Cloud CLI command:

gcloudprojectsadd-iam-policy-bindingPROJECT_NUMBER\--memberserviceAccount:service-PROJECT_NUMBER@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com\--roleroles/bigquerydatatransfer.serviceAgent

ReplacePROJECT_NUMBER with the project number associatedwith this service account.

Error:There was an error loading this table. Check that the table exists and that you have the correct permissions.

Resolution:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. Click the destination dataset used in the transfer.

3. Click theSharing menu, and then clickPermissions.

4. Expand theBigQuery Data Editor role.

5. Verify that the BigQuery Data Transfer Service service agent is added to thisrole. If not, grant the BigQuery Data Editor (roles/bigquery.dataEditor)role to the BigQuery Data Transfer Service service agent.

Error:A permission denied error was encountered: PERMISSION_DENIED. Please ensure that the user account setting up the transfer config has the necessary permissions, and that the configuration settings are correct

Resolution:

1. In the Google Cloud console, go to theData Transfers page.

   Go to Data Transfers

2. Click the failed transfer, then select theConfiguration tab.

3. Verify that the transfer owner listed in theUser field has all therequired permissions for the data source.

If the transfer owner does not have all the required permissions, grant therequired permissions byupdating their credentials.You can also change the transfer owner to another user with the requiredpermissions.

Error:Authentication failure: User Id not found. Error code: INVALID_USERID

Resolution: The transfer owner has an invalid user ID. Change the transferowner to a different user byupdating their credentials.If you are using a service account, you should also verify that the accountsrunning the data transfer have all therequired permissions to use a service account.

Error:The user does not have permission

Resolution: Verify that the transfer owner is a service account, and thatthe service has all the required permissions set. Another possibility is thatthe service account used was created under a different project than the projectused to create this transfer. To resolve cross-project permission issues, seethe following resources:

• Enable service accounts to be attached across projects
• Cross-project Service Account Authorization (for granting the necessary permissions)

Error:HttpError 403 when requesting returned "The caller does not have permission"

googleapiclient.errors.HttpError: <HttpError 403 when requesting returned "The caller does not have permission". Details: "The caller does not have permission">

This error might appear when you attempt to set up a scheduled query with aservice account.

Resolution: Ensure that the service account has all thepermissions required to schedule or modify a scheduled query,and ensure that the user setting up the scheduled query hasaccess to the service account.

If the correct permissions are all assigned but you still encounter theerror, check if theDisable Cross-Project Service Account Usagepolicy is enforced on the project by default. You can check for the policy inthe Google Cloud console by navigating toIAM & Admin>Organization Policies and searching for the policy.

If theDisable Cross-Project Service Account Usage policy is enforced,you can disable the policy by doing the following:

1. Identify the service accounts associated with the project using the Google Cloud console,by navigating toIAM & Admin>Service Accounts. Thisview displays all service accounts for the current project.
2. Disable the policy in the project where the service accounts are locatedusing the following command. To disable this policy, the user must beanOrganization Policy Administrator.Only the Organization Administrator can grant a user this role.

gcloudresource-managerorg-policiesdisable-enforceiam.disableCrossProjectServiceAccountUsage--project=[PROJECT-ID]

Event-driven transfer configuration issues

The following are common issues you might encounter when creating anevent-driven transfer.

Error:Data Transfer Service is not authorized to pull message from the provided Pub/Sub subscription.

Resolution: Verify that the BigQuery Data Transfer Serviceservice agent is grantedthepubsub.subscriber role:

1. In the Google Cloud console, go to thePub/Sub page.

   Go to Pub/Sub

2. Select the Pub/Sub subscription that you used in the event-driventransfer.

3. If the info panel is hidden, clickShow info panel in the upper rightcorner.

4. In thePermissions tab, verify that the BigQuery Data Transfer Serviceservice agenthas thepubsub.subscriber role

If the service agent doesn't have thepubsub.subscriber role granted. Clickperson_addAdd principal to grant thepubsub.subscriber role toservice-PROJECT_NUMBER@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com

Error:Cloud Pub/Sub API has not been used in projectPROJECT_NUMBER before or it is disabled.

Resolution: Verify that Cloud Pub/Sub API is enabled for your project:

1. In the Google Cloud console, go to theAPIs & Services page.

   Go to APIs & Services

2. ClickEnable APIs and services.

3. Search forCloud Pub/Sub API, select the first result and clickEnable.

Error:Data Transfer Service does not have required permission to use project quota of projectPROJECT_NUMBER to access Pub/Sub.

Resolution: Verify that the BigQuery Data Transfer Serviceservice agent is grantedtheserviceusage.serviceUsageConsumer role:

1. In the Google Cloud console, go to theIAM & Admin page.

   Go to IAM & Admin

2. Select theInclude Google-provided role grants checkbox.

3. Verify that the service account with the nameservice-<project_number>@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.comis shown and that it has been granted theService Usage Consumer role.

   

Issue: When you use Cloud Storage event-driven transfer, no transfer run is triggered after uploading or updating files in Cloud Storage bucket.

Transfer runs are not triggered immediately after an event is received. Itmight take several minutes to trigger a transfer run. To check the status of thenext transfer run, you can check theTarget date for next run field in therun history.This field displays the schedule time for the next run, or displayswaiting for events to schedule next run if no events were received. If youhave uploaded or updated files in your Cloud Storage bucket, but theTarget date for next run has not updated and no runs are triggered for 10-20minutes, see the following resolution.

Resolution: Verify that your Pub/Sub subscription specified in thetransfer config is able to get messages published from Cloud Storage events:

1. In the Google Cloud console, go to thePub/Sub page.

   Go to Pub/Sub

2. Select the Pub/Sub subscription that you used in the event-driventransfer.

3. In theMetrics tab, check the "Oldest unacked message age" graph andsee if there are any messages.

If no messages are published, check if Pub/Sub notification iscorrectly configured for Cloud Storage. You may use the following Google Cloud CLIcommand to check the notification configurations associated with your bucket:

gcloudstoragebucketsnotificationslistgs://BUCKET_NAME

ReplaceBUCKET_NAME with the name of the bucket that youuse for notification. For information on configuring a Pub/Sub notificationfor Cloud Storage, seeConfigure Pub/Sub notification for Cloud Storage.

If there are messages, check if the same Pub/Sub subscription is usedin another event-driven transfer configs. The same Pub/Sub subscriptioncannot be reused by multiple event-driven transfer configs. For more informationabout event-driven transfers, seeEvent-driven transfers.

Amazon S3 transfer issues

The following are common errors encountered whencreating an Amazon S3transfer.

Amazon S3PERMISSION_DENIED errors

Error:The AWS Access Key Id you provided does not exist in our records.

Resolution: Verify that the access key exists and the ID is correct.

Error:The request signature we calculated does not match the signature you provided. Check your key and signing method.

Resolution: Verify that the transfer configuration has the correct corresponding Secret Access Key

Error:Failed to obtain the location of the source S3 bucket. Additional details: Access Denied

Error:Failed to obtain the location of the source S3 bucket. Additional details: HTTP/1.1 403 Forbidden

Error:Access Denied (S3 error message)

Resolution: Ensure the AWS IAM user has permission to perform thefollowing:

• List the Amazon S3 bucket.
• Get the location of the bucket.
• Read the objects in the bucket.

Error:Server unable to initialize object upload.; InvalidObjectState: The operation is not valid for the object's storage class

Error:Failed to obtain the location of the source S3 bucket. Additional details: All access to this object has been disabled

Resolution: Restore any objects that are archived to Amazon Glacier. Objects in Amazon S3 that are archived to Amazon Glacier are not accessible until they are restored

Error:All access to this object has been disabled

Resolution: Confirm that the Amazon S3 URI in the transfer configuration is correct

Amazon S3 transfer limit errors

Error:Number of files in transfer exceeds limit of 10,000.

Resolution: Evaluate if the number ofwildcardsin the Amazon S3 URI can be reduced to just one. If this is possible, retry witha new transfer configuration, as themaximum number of files per transfer run will be higher.You can also evaluate if the transfer configuration can be split into multipletransfer configurations, each transferring a portion of the source data.

Error:Size of files in transfer exceeds limit of 16492674416640 bytes.

Resolution: Evaluate if the transfer configuration can be split into multiple transfer configurations, each transferring a portion of the source data.

General Amazon S3 issues

Issue: Files are transferred from Amazon S3 but not loaded into BigQuery.

The transfer logs may look similar to the following:Moving data from Amazon S3 to Google Cloud complete: MovedN object(s).No new files found matchingAmazon_S3_URI.

Resolution: Confirm that the Amazon S3 URI in the transfer configurationis correct. If the transfer configuration was meant to load all files with acommon prefix, ensure that the Amazon S3 URI ends with a wildcard. Forexample, to load all files ins3://my-bucket/my-folder/, the Amazon S3 URIin the transfer configuration must bes3://my-bucket/my-folder/*, not justs3://my-bucket/my-folder/.

Azure Blob Storage transfer issues

The following are common errors encountered whencreating anBlob Storage transfer.

Error:Number of files in transfer exceeds the limit of 10,000.

Resolution: Reduce the number ofwildcardsin the Blob Storage data path to 0 or 1, so the file limitincreases to 10,000,000. You can also split into multiple transferconfigurations, each transferring a portion of the source.

Error:Size of files in transfer exceeds the limit of 15 TB.

Resolution: Split into multiple transfer configurations, each transferringa portion of the source data.

Error:Provided Azure SAS Token does not have required permissions.

Resolution: Verify that the Azure SAS token in the transfer configuration iscorrect. For more information, seeShared access signature (SAS).

Error:Transfer encountered error, status:PERMISSION_DENIED, details:[This request is not authorized to perform this operation.]

Resolution: Verify that the IP ranges used by BigQuery Data Transfer Serviceworkers are added to your list of allowed IPs. For more information, seeIP restrictions.

Issue: Files are transferred from Blob Storage but not loaded into BigQuery.

The transfer logs might look similar to the following:Moving data to Google Cloud complete: Moved<var>N</var> object(s). No new files found matching Blob Storage data path.

Resolution: Verify that the Blob Storage data path in thetransfer configuration is correct.

Campaign Manager transfer issues

The following are common errors encountered whencreating a CampaignManager transfer.

Error:Import failed - no data was available for import. Please verify that data existence was expected.

Error:No data available for the requested date. Please try an earlier run date or verify that data existence was expected.

Resolution: Verify you are using thecorrect IDfor the transfer. If you areusing the correct ID, verify the Campaign Manager Cloud Storage bucketcontains Data Transfer V2.0 files for the specified date range. If the filesexist, schedule a backfill for the affected date range. For more information oncreating a Campaign Manager backfill request, seeManually trigger a transfer or backfill.

You can verify whether the files existed when the transfer run was scheduledby checking the created time of the files in the Cloud Storage bucket. Insome cases, the first transfer run of the day may be scheduled prior to thegeneration of the first batch of Campaign Manager Data Transfer files.Subsequent runs on the same day and the following day will load all the filesgenerated by Campaign Manager.

Error:A permission denied error was encountered: PERMISSION_DENIED. Please ensure that the user account setting up the transfer config has the necessary permissions, and that the configuration settings are correct.

Resolution: The user creating the Campaign Manager transfermust have read access to theCloud Storage bucketcontaining the Data Transfer V2.0 files. You can obtain information about theCloud Storage bucket and request access from your Campaign Manageradministrator.

Google Ads transfer issues

The following are common errors encountered whencreating aGoogle Ads transfer.

Issue: Transfer run succeeds but some accounts aren't showing up in the destination table.

Resolution: There could be many reasons that a account doesn't show up in report, some common causes are:

• This can occur if there isn't any report activity for the requested day, resulting in no data rows being generated.

• This can also occur if your Google Ads account is inactive, orCANCELLED. The Google Ads API doesn't support queries on inactive accounts, so the Google Ads connector filtered out inactive accounts from transfer run. To reactivate your Google Ads account, seeReactivate a canceled Google Ads account.

Error:AUTH_ERROR_TWO_STEP_VERIFICATION_NOT_ENROLLED

Resolution: This error indicates that the Google Ads user used in this transfer does not have 2-step verification enabled. For information about enabling 2-step verification, seeTurn on 2-Step Verification.

Error:No jobs to start for run

Resolution: This error indicates there is no load job started for the transfer run due to an invalid setup or errors that occurred during processing. Follow the steps to resolve this error:

• Check the transfer run history log for errors and warnings.
• If the tables you are trying to load don't have columnssegments_date,segments_week,segments_month,segments_quarter, orsegments_year, it's expected to skip loading whenrun_date is not the latest date. SeeManually trigger a Google Ads transfer andCustom reports for more details.
• Request support if you have trouble finding the root cause.

Error:Import failed - no data was available for import. Please verify that data existence was expected.

Error:No data available for the requested date. Please try an earlier run date or verify that data existence was expected.

Resolution: If you receive this error when you are creating aGoogle Ads transfer,request supportand include a screen capture of the error message.

Error:AuthenticationError.NOT_ADS_USER.

Resolution: The user setting up the Google Ads transfermust have a Google Ads account/login.

Error:Request is missing required authentication credential

Resolution: The user or the service account doesn't have access to the Ads account. FollowRequired permissions to grant the user or the service account the required permissions.

Error:ERROR_GETTING_RESPONSE_FROM_BACKEND.

Resolution: If a Google Ads transfer run fails and returnsERROR_GETTING_RESPONSE_FROM_BACKEND,enable theExclude Removed/Disabled Items optionin the transfer configuration andset up a backfillto attempt to retrieve data for the days impacted by the failed transfer run.

Warning:Data for the report ClickStats was not available for the specified date.

Error:INVALID_DATE_RANGE_FOR_REPORT.

Resolution: This is expected when backfillingClick Performance Report data for more than 90 days back. In this case, you will see this warning or error and theClickStats table won't be updated for the specified date.

Error:Error while processing report for tabletable_name for account idaccount_id. Http(400) Bad Request;

Error:AuthorizationError.TWO_STEP_VERIFICATION_NOT_ENROLLED

Resolution: If the user account associated with this transfer does nothave 2-step verification (or multi-factor authentication) enabled,enable 2-step verificationfor this account and then rerun the failed transfer job. Service accounts areexempted from the 2-step verification requirement.

Error:Quota exceeded: Your project exceeded quota for imports per project

Resolution: Transfers are subject to BigQueryquotas on load jobs. If you hit the quota limit for load job, try to reduce unnecessaryloadings by usingtable_filter, deleting unused transfer configs or reducing the refresh window.If you need to increase your quota, contact your Google Cloud sales representative. For moreinformation, seeQuotas and limits.

Google Ad Manager transfer issues

The following are common errors encountered when creating aGoogle Ad Managertransfer.

Error:Another transfer run is concurrently processing table.

Resolution: This error may occur when another transfer run with the same transfer configuration hasn't finished running when the current transfer run begins.

• If the latency is caused by large Google Ad Manager Data Transfer (Google Ad Manager DT) files, consider enablingparquet format which has better loading performance.
• If the latency is caused by match tables, consider skipping match tables by setting parameterload_match_tables tofalse.
• Adjust the repeat frequency if the transfer consistently takes a longer time to process than the current repeat frequency.

Error:No data available for the requested date. Please try an earlier run date or verify that data existence was expected.

Error:Import failed - no data was available for import. Please verify that data existence was expected.

Resolution: Verify the Google Ad ManagerCloud Storage bucketcontains data transfer files for the specified date range. Your Google Ad Manageradministrator manages theCloud Storage bucketcontaining your Data Transfer files. Users creating Google Ad Manager transfersmust be members of the Google Group with read access to the bucket.

You can verify Cloud Storage permissions by attempting to read files in theGoogle Ad ManagerData Transfer bucket.For more information on Google Ad Manager Cloud Storage buckets, seeAccess Google Ad Manager storage buckets.

You can verify whether the files existed when the transfer run was scheduledby checking the created time of the files in the Cloud Storage bucket. Insome cases, the first transfer run of the day may be scheduled prior to thegeneration of the first batch of Google Ad Manager Data Transfer files.Subsequent runs on the same day and the following day will load all the filesgenerated by Google Ad Manager.

If the files exist in the Data Transfer bucket and you have read permissions,schedule a backfill for the affected date range. For more information oncreating a Google Ad Manager backfill request, seeSet up a backfill.

Error:AuthenticationError: NO_NETWORKS_TO_ACCESS.

Resolution: Ensure you have read access to the Google Ad Managernetwork. If you need assistance determining network access, contactGoogle Ad Manager support.

Error:Error code 9 : Fieldfield_name?field_name?field_name?RefererURL is unknown.; Table:table_name

Resolution: Ensure you are not using the thorn (þ) delimiter. The thorndelimiter is unsupported. Use of the thorn is indicated by the ? inthe error message.

Error:Incompatible table partitioning specification. Destination table exists with partitioning specification interval(type:Day,field:) clustering

Resolution: The Google Ads Manager connector does not support transfer data to a dataset with clustering. Use a dataset without clustering instead.

Google Merchant Center transfer issues

The following are common errors encountered whencreating a Google MerchantCenter transfer.

Error:No data to transfer found for the Merchant account. If you have just created this transfer - you may need to wait for up to 90 minutes before the data of your Merchant account are prepared and available for the transfer.

Resolution: You receive this error if you set up a transfer using thedefault starting date and time in the Schedule section. If you use the defaultscheduling values, the first transfer run starts immediately after the transferis created, but it fails because your Merchant account data must be preparedbefore it can be transferred. Wait 90 minutes and thenset up a backfillfor today, or you can wait until tomorrow for the next scheduled run.

Error:No data to transfer found for Merchant account. This can be because your account currently doesn't have any products.

Resolution: This error indicates that your Merchant account has noproducts. The transfer will begin working once you add products to your Merchantaccount.

Error:Transfer user doesn't have access to the Merchant account. Please verify access in the Users section of the Google Merchant Center.

Resolution: This error indicates that the user who set up the transferdoesn't have access to the Merchant account used by the transfer. Toresolve the issue,verify and grant missing account accessin the Google Merchant Center.

Error:Transfer user doesn't have user roles that allows access to the product data of the Merchant account. Please verify access and roles in the Users section of the Google Merchant Center.

Resolution: This error indicates that the user who set up the transferdoesn't have access to the product data of the Merchant account used by thetransfer. To resolve the issue,verify and grant missing user rolesin the Google Merchant Center.

Error:Historical backfills are not supported.

Resolution: You receive this error if youset up a backfillfor previous days. This error is expected. Historical backfills are not supported.You can only set up a backfill for today torefresh today's data after the scheduled daily run has already finished.

Google Play transfer issues

The following are common errors encountered whencreating a Google Playtransfer.

Error:No jobs to start for run

Resolution: Verify that the user hassufficient permissions to initiatea Google Play transfer, thenverify if the user has specified the correct Cloud Storage bucket whensetting up the transfer. If the user is still experiencing the error, you canrequest support.

Issue: Revenue and financial reports did not load into BigQuery

Resolution: To access financial Google Play reports, users must have theView financial data permission. To manage developeraccount permissions, seeAdd developer account users and manage permissions.

HubSpot transfer issues

Error:PERMISSION_DENIED: Permission denied. Your Access Token may lack required access to the provided account. Please also check for typos like whitespace or if the provided accountId even exists

Resolution: Check that your private app access token is correct, theHubSpot user has theSuper Admin role, and the private apphas all the required scopes. For more information, seeHubSpotprerequisites.

Error:INVALID_ARGUMENT: Table 'NAME' does not exist in asset "ASSET"

Resolution: Check that the specified asset name is valid and does notcontain any leading or trailing spaces. Whencreating aHubSpottransfer, werecommend that you clickBrowse to select the asset from the list ofavailable objects.

Error:FAILED_PRECONDITION: Rate limit exceeded.

Resolution: The HubSpot API rate limit has been exceeded.Wait for some time before retrying the data transfer. You can also considerreducing the frequency of data transfer jobs and limiting simultaneoustransfers for similar accounts.

Error:UNAUTHENTICATED: Authentication failed. Please verify your HubSpot access token.

Resolution: Check that your private appaccess token is correct. For more information, seeHubSpot prerequisites.

Error:UNKNOWN: An unknown error occurred while processing the request.

Resolution: Verify that the HubSpot private app accesstoken is correct and that the required permissions are in place for theobjects being accessed, then retry the transfer job.

Klaviyo transfer issues

The following are common issues you might encounter whencreating a Klaviyo transfer.

Error:PERMISSION_DENIED: Permission denied. Your API key may lack required access scopes

Resolution: Verify that the Klaviyo private API key has atleast theREAD ONLY access level. For more information, seeKlaviyoprerequisites.

Error:FAILED_PRECONDITION

Resolution: Retry the transfer with a shorter date range.

Error:UNKNOWN: An unknown error occurred while processing the request.

Resolution: Verify that the API key of the account is valid, then retrythe transfer job.

Error:INTERNAL: An unknown error occurred while processing the request.

Resolution: Verify that the API key of the account is valid, then retrythe transfer job.

Microsoft SQL Server transfer issues

The following are common issues that you might encounter whencreating a Microsoft SQL Server transfer.

Error:FAILED PRECONDITION: A TLS/SSL handshake error occurred: unable to find valid certification path to requested target. Please check your TLS/SSL configuration and certificate validity.

Resolution: Verify the validity of your certificate with the followingsteps:

1. Replace the SSL/TLS certificate on SQL Server with acertificate issued by a trusted Public Certificate Authority. For more information, seeTLS configuration.
2. Verify that the new certificate includes the complete certificate chain,including all intermediate and root certificates.
3. After updating the certificate, restart the SQL Serverservice to apply the new SSL/TLS configuration.
4. Once the server has restarted, create the transfer configuration again toverify that the updated certificate is trusted and the TLS/SSL handshakecompletes successfully.

Error:INVALID_ARGUMENT: Catalog 'SCHEMA' does not exist in asset "SCHEMA/TABLE"

Resolution: Verify that the catalog and table are present in theSQL Server and spelled correctly. Also verify thatthe user has sufficient access permissions to browse or query the catalog andits contents.

Mailchimp transfer issues

The following are common issues you might encounter whencreating a Mailchimp transfer.

Error:INVALID_ARGUMENT: Invalid request. Please check the input parameters (Credentials, Table, etc.) and try again.

Resolution: Check that the specified asset name is valid and does notcontain any leading or trailing spaces. Whencreating aMailchimp transfer, werecommend that you clickBrowse to select the asset from the list ofavailable objects.

Error:PERMISSION_DENIED: Permission denied. Your credentials may lack required access.

Resolution: Check that the Mailchimp user has all therequired permissions. TheAdmin userlevel inMailchimp has minimum level of access to transfer allMailchimp objects.

Error:FAILED_PRECONDITION: Operation failed due to precondition violation (ex- Rate limit exceeded, Server Error). Please try again later.

Resolution: Wait for some time before retrying the data transfer. You canalso checkMailchimp statusfor any service outages.

Error:UNKNOWN: An unknown error occurred while processing the request.

Resolution: Confirm that the API Key is valid and that the user has allthe required permissions, then retry the data transfer job.

MySQL transfer issues

The following are common issues you might encounter whencreating a MySQL transfer.

Error:PERMISSION_DENIED. Failed to authenticate or permission denied with the provided credentials when starting to transfer assetasset-name.

Resolution: Check if theconnector.authentication.usernameandconnector.authentication.password parameters that youprovided are valid and functional.

Error:NOT_FOUND. Invalid data source configuration provided when starting to transfer assetasset-name: APPLICATION_ERROR;google.cloud.bigquery.federationv1alpha1/ConnectorService.StartQuery;INVALID_ARGUMENT:Exception was thrown by the Connector implementation: Tabletable-name does not exist in assetasset-name.

Resolution: Check that the spelling of the table or viewname is correct, the referenced table or view name exists, and the synonym (alias)points to an existing table or view.

If the table or view exists, ensure that the correct access privileges aregranted to the database user requiring access to the table. If the table orview doesn't exist, create the table.

If you are attempting to access a table or view in another schema, ensurethat the correct schema is referenced and that access to the object is granted.

If a table or view name is provided, ensure that it is specified asobject_name;otherwise, it is blank.

Error:SERVICE_UNAVAILABLE. Timed out when starting to transfer assetasset-name. Ensure the datasource is reachable and the datasource configuration (Credentials, Network Attachment etc.) is correct.

Error:DEADLINE_EXCEEDED. Timed out when starting to transfer assetasset-name. Ensure the datasource is reachable and the datasource configuration (Credentials, Network Attachment etc.) is correct.

Resolution: Check if the database details provided are correct, and checkthat the network attachment used for the transfer configuration is set upcorrectly. It's also possible that the transfer didn't finish within the deadline.

Error:INTERNAL

Resolution: Something else caused the transfer to fail.Contact Cloud Customer Care for help resolvingthis issue.

Error:INVALID_ARGUMENT. Connection to the host and port failed. Please check that the host, port, encryptionMode and network attachment are correct.

Resolution: Ensure the host, port, encryption mode, and network settingsare correctly configured. Verify network connectivity and that the databaseserver is accessible. IfEncryptionMode is set toFULL, confirm that theserver supports the required protocols, has a valid certificate, and allowssecure connections. IfEncryptionMode is set toDISABLE, check that theserver permits non-SSL connections. Review the application and database logsfor connection-related or SSL/TLS-related errors.

Oracle transfer issues

The following are common issues you might encounter whencreating an Oracle transfer.

Error:PERMISSION_DENIED. ORA-01017: invalid username/password; logon denied

Resolution: Check that the provided Oracle credentials arevalid.

Error:PERMISSION_DENIED. ORA-01045: user lacks CREATE SESSION privilege; logon denied

Resolution: Grant theCREATE SESSION system privileges to the Oracleuser. For more information about granting Oracle privileges, seeGRANT.

Error:SERVICE_UNAVAILABLE. ORA-12541: Cannot connect. No listener at hostHOSTNAME portPORT orSERVICE_UNAVAILABLE. Connection failed: IO Error. The Network Adapter could not establish the connection

Resolution: Check that the provided Hostname and Port details are correct,and that the network attachment is set up correctly.

Error:NOT_FOUND. ORA-00942: table or view does not exist

Resolution: Check each of the following:

• The spelling of the table or view name is correct.
• The referenced table or view name exists.
• The synonym points to an existing table or view. If the table or view does exist, ensure that the correct access privileges are granted to the database user requiring access to the table. Otherwise, create the table.
• if you are attempting to access a table or view in another schema, make sure that the correct schema is referenced and that access to the object is granted.

Error:NOT_FOUND. Schemaschema does not exist.

Resolution: The specified schema does not exist.

Error:DEADLINE_EXCEEDED

Resolution: The transfer run did not finish within the six hour maximumduration deadline. Reduce your transfer run times by splitting up largetransfers into multiple smaller ones.

Error:INTERNAL

Resolution: Something else caused the transfer to fail.Contact Cloud Customer Care for help resolvingthis issue.

Error:INVALID_ARGUMENT

Resolution: A supplied value in the transfer configuration is invalid,causing the transfer to fail. For information about valid transfer configurationvalues, seeSet up an Oracle transfer.

Error:SQL Error [1950] [42000]: ORA-01950: no privileges on tablespace 'TablespaceName'

Resolution: Assign the default tablespace to the user. For moreinformation, seeAssigning a Default Tablespace.

Error:403 PERMISSION_DENIED. Required 'compute.subnetworks.use' permission forproject

Resolution: This error can occur if your network attachment is located ina different project than where the transfer configuration is located. To remedythis issue, you must grant the service account (for example,service-customer_project_number@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com)the following permissions in the project where the network attachment is located:

• compute.networkAttachments.get
• compute.networkAttachments.update
• compute.subnetworks.use
• compute.regionOperations.get

This error can also occur if the network attachment is trying to connect to aVirtual Private Cloud (VPC) that is located in a different project, such as a shared VPC.In this case,you must grant the service account (for example,service-customer_project_number@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com)thecompute.subnetworks.use permission on the hosting project of the shared VPC.

PayPal transfer issues

The following are common issues that you might encounter whencreating a PayPal transfer.

Error:PERMISSION_DENIED: Authorization failed due to insufficient permissions.

Resolution: Verify that the client and the client secret have therequiredpermissions to accessthe PayPal object that is being transferred.

Error:INVALID_ARGUMENT: Table 'OBJECT' does not exist in asset "OBJECT"

Resolution: Verify that the object name is valid.

Error:INVALID_ARGUMENT: The given start date 'DATE' cannot be parsed. Please provide it in 'yyyy-MM-dd' format.

Resolution: Verify that the start date is in correct syntax.

Error:UNAUTHENTICATED: Please provide a valid clientId and client secret.

Resolution: Verify that the client ID and client secret are correct.

Error:UNKNOWN: An unknown error occurred while processing the request.

Resolution: Verify that the secret key and account ID are valid and thattherequiredpermissions are in placefor the selected objects, and then retry the transfer job. If the issuepersists,contact Cloud Customer Care.

Error:INTERNAL: An unknown error occurred while processing the request.

Resolution: Verify that the secret key and account ID are valid and thattherequiredpermissions are in placefor the selected objects, and then retry the transfer job. If the issuepersists,contact Cloud Customer Care.

PostgreSQL transfer issues

The following are common issues you might encounter whencreating a PostgreSQL transfer.

Error:PERMISSION_DENIED. Failed to authenticate or permission denied with the provided credentials when starting to transfer assetasset-name.

Resolution: Check if theconnector.authentication.usernameandconnector.authentication.password parameters that youprovided are valid and functional.

Error:NOT_FOUND. Invalid data source configuration provided when starting to transfer assetasset-name: APPLICATION_ERROR;google.cloud.bigquery.federationv1alpha1/ConnectorService.StartQuery;INVALID_ARGUMENT:Exception was thrown by the Connector implementation: Tabletable-name does not exist in assetasset-name.

Resolution: Check that the spelling of the table or viewname is correct, the referenced table or view name exists, and the synonym (alias)points to an existing table or view.

If the table or view exists, ensure that the correct access privileges aregranted to the database user requiring access to the table. If the table orview doesn't exist, create the table.

If you are attempting to access a table or view in another schema, ensurethat the correct schema is referenced and that access to the object is granted.

If a table or view name is provided, ensure that it is specified asobject_name;otherwise, it is blank.

Error:SERVICE_UNAVAILABLE. Timed out when starting to transfer assetasset-name. Ensure the datasource is reachable and the datasource configuration (Credentials, Network Attachment etc.) is correct.

Error:DEADLINE_EXCEEDED. Timed out when starting to transfer assetasset-name. Ensure the datasource is reachable and the datasource configuration (Credentials, Network Attachment etc.) is correct.

Resolution: Check if the database details provided are correct, and checkthat the network attachment used for the transfer configuration is set upcorrectly. It's also possible that the transfer didn't finish within the deadline.

Error:INTERNAL

Resolution: Something else caused the transfer to fail.Contact Cloud Customer Care for help resolvingthis issue.

Error:INVALID_ARGUMENT. Connection to the host and port failed. Please check that the host, port, encryptionMode and network attachment are correct.

Resolution: Ensure the host, port, encryption mode, and network settingsare correctly configured. Verify network connectivity and that the databaseserver is accessible. IfEncryptionMode is set toFULL, confirm that theserver supports the required protocols, has a valid certificate, and allowssecure connections. IfEncryptionMode is set toDISABLE, check that theserver permits non-SSL connections. Review the application and database logsfor connection-related or SSL/TLS-related errors.

Salesforce transfer issues

The following are common errors encountered whencreating a Salesforce transfer.

Error:Permission Denied: invalid_client. invalid client credentials

Resolution: Verify that the ClientSecret provided is valid.

Error:Permission Denied: invalid_client. client identifier invalid

Resolution: Verify that the ClientId provided is valid.

Error:Permission Denied: Error encountered while establishing connection

Resolution: Check if the Salesforce MyDomain Name provided is correct.

Error:NOT_FOUND. asset typeasset_name is not supported. If you are attempting to use a custom object, be sure to append the "__c" after the entity name. Please reference your WSDL or use the describe call for the appropriate names.

Resolution: Follow the guidance in the error code, and verify that the provided asset name is correct.

Error:SERVICE_UNAVAILABLE

Resolution: The service is temporarily unable to handle the request. Waita few minutes and try the operation again.

Error:DEADLINE_EXCEEDED

Resolution: The transfer run did not finish within the six hour maximumduration deadline. Minimize your transfer run times by splitting up largetransfers into multiple smaller ones.

Error:Failed to create recordReader to read partition : Batch failed. BatchId='batch_id', Reason='FeatureNotEnabled : Binary field not supported'

Resolution: The connector doesn't support sObject data structures thatcontain binary fields. Remove sObject data structures that contain binaryfields from your transfer jobs. For more information, seeError 'Batch failed: FeatureNotEnabled: Binary field not supported' when you export related objectin the Salesforce documentation.

Error:RESOURCE_EXHAUSTED: PrepareQuery failed : ExceededQuota : ApiBatchItems Limit exceeded

Resolution: This error appears when you have exceeded the dailyApiBatchItems API limitfor job runs. Salesforce has a daily API limit that is resetevery 24 hours. To resolve this error, we recommend that you split up andschedule your transfer runs so that it doesn't exceed the daily batch API limit.You can also contact Salesforce support to increase your dailylimit.

Error:Permission Denied: invalid_grant. no client credentials user enabled

Resolution: Verify that theRun as field in theClient Credentials Flowsection of the Salesforce Connected App contains the correctusername. For more information, seeCreate a Salesforce Connected App.

Error:FAILED_PRECONDITION: BatchId='batch-id', Reason='InvalidBatch : Failed to process query: OPERATION_TOO_LARGE: exceeded 100000 distinct ids'

Resolution: Ensure the user's profile does not impose restricted querylimits for the extracted sObject. If the issue persists, use the credentials ofa Salesforce user with System Administrator privileges tocomplete the extraction.

Error:FAILED_PRECONDITION: Batch failed. BatchId='batch-id', Reason='InvalidBatch : Failed to process query: TXN_SECURITY_NO_ACCESS: The operation you requested isn't allowed due to a security policy in your organization. Contact your administrator for more information about security policies.

Resolution: Ensure that the user's profile or permission set includes therequired object and field-level permissions for the extracted sObject. Contactyour administrator to update these permissions or assign a role with therequired access.

Error:FAILED_PRECONDITION: Cannot establish connection to Salesforce to describe SObject: 'SObject_Name' due to error: TotalRequests Limit exceeded., Cause:null Retry after some time post quota reset.

Resolution: This error can occur when you haveexceeded yourlimit for load jobs. Wait untilyour quota resets before trying again.

Error:FAILED_PRECONDITION: There was an issue connecting to Salesforce Bulk API.

Resolution: This error can occur when you include a network attachmentwith your transfer but have not configured your public NAT and set up your IPallow list. To resolve this error, do all the steps inSetup IP allowlist forSalesforcetransfers.

Error:SYSTEM ERROR when starting to transfer assetasset_name, try again later.

Resolution: This error can occur when the Bulk API isn't enabledfor your Salesforce organization. To resolve this error, update yourSalesforce license to include Bulk API support. For moreinformation, seeEnable Asynchronous API.

Shopify transfer issues

Error:PERMISSION_DENIED: Permission denied. Your API key may lack required access to the provided account. Please also check for typos like whitespace or if the provided accountId even exists

Resolution: Verify that the Shopify Admin API access tokenis correct, and verify that the Shopify app has all therequired accessroles.

Error:INVALID_ARGUMENT: Table 'NAME' does not exist in asset "ASSET"

Resolution: Check that the specified asset name is valid and does notcontain any leading or trailing spaces. Whencreating aShopifytransfer, werecommend that you clickBrowse to select the asset from the list ofavailable objects.

Error:UNAUTHENTICATED: Authentication failed. Please verify your Shopify access token.

Resolution: Check that your Shopify Admin API access tokenis correct. For more information, seeShopifyprerequisites.

Error:UNKNOWN: An unknown error occurred while processing the request.

Resolution: Verify that the Shopify Admin API access tokenand the shop name is correct, and then retry the transfer job. If the issuepersists,contact Cloud Customer Care.

Stripe transfer issues

The following are common errors encountered whencreating a Stripe transfer.

Error:PERMISSION_DENIED: Permission denied. Your API key may lack required access to the provided account. Please also check for typos like whitespace or if the provided accountId even exists

Resolution: If you are using a restricted API key, check that the key has permission to access the Stripe object being transferred. Once you have updated the permissions on the restricted key, try running the data transfer again. For information about managing secret keys, seeSecret and restricted keys.

Error:INVALID_ARGUMENT: Table 'NAME' does not exist in asset "ASSET"

Resolution: Check that the specified asset name is valid and does not contain any leading or trailing spaces. Whencreating a Stripe transfer, we recommend that you clickBrowse to select the asset from the list of available objects.

Error:UNAUTHENTICATED: Authentication failed. Please verify your Stripe API key.

Resolution: Check that your secret key and account ID are correct. For information on how to retrieve this information, seeStripe prerequisites.

Error:RESOURCE_EXHAUSTED: Rate limit exceeded.

Resolution: You can exceeded the Stripe API rate limit. Wait for some time before retrying the data transfer job request. To prevent this issue, consider reducing the frequency of data transfer jobs and limiting concurrent transfers from the same account.

Error:UNAVAILABLE: Stripe service is temporarily unavailable. Please try again shortly.

Resolution: Wait for some time before retrying the data transfer. You can checkStripe status for any service outages.

Error:UNKNOWN: An unknown error occurred while processing the request.

Resolution: Verify that the secret key and account ID are valid and that the required permissions are in place for the selected objects, and then retry the transfer job. If the issue persists,contact Cloud Customer Care.

Error:INTERNAL: An unknown error occurred while processing the request.

Resolution: Verify that the secret key and account ID are valid and that the required permissions are in place for the selected objects, and then retry the transfer job. If the issue persists,contact Cloud Customer Care.

ServiceNow transfer issues

The following are common issues you might encounter whencreating a ServiceNow transfer.

Error:UNAUTHENTICATED. Required authentication credentials were not provided when starting to transfer assetasset-name.

Resolution: Verify that the credentials provided (username,password,ClientID andClient Secret) are valid, correctly configured, and not expired.

Error:INVALID_ARGUMENT. Invalid datasource configuration provided when starting to transfer asset

Error:INVALID_ARGUMENT: Http call to ServiceNow instance returned status code 400.. Please make sure the instance/endpoint provided exists/is correct.

Resolution: Verify that the ServiceNow instance URL and API endpoint are correct, and the referenced table exists and is accessible with the provided credentials.

Error:PERMISSION_DENIED. User credentials don't have permission or are invalid for accessing the ServiceNow asset or API.

Resolution: Verify that the credentials provided are correct, and that the ServiceNow account has sufficient permissions to access the specified assets or tables.

Error:UNAVAILABLE. ServiceNow instance is temporarily unreachable or experiencing downtime.

Resolution: Verify that the network connectivity is stable and try again after some time. This error is likely due to a temporary ServiceNow outage or transient connectivity issue.

Error:RESOURCE EXHAUSTED. ServiceNow API rate limit or quota has been exceeded, or operations are too large.

Resolution: You have exceeded the ServiceNow API quota or rate limit. Consider reducing the volume or frequency of requests and try again.

Error:FAILED_PRECONDITION: There was an issue connecting to API.

Resolution: This error can occur when you include a network attachmentwith your transfer but have not configured your public NAT and set up your IPallow list. To resolve this error,create a network attachmentby defining a static IP address.

Teradata transfer issues

The following are common issues you might encounter when creating a Teradatatransfer.

Error:Skipping extraction since table does not have change tracking column.

Resolution: The preceding message might appear when you attempt to performa Teradata transfer on an already migrated table using anexisting on-demand transfer config. If you want to start a newtransfer on an already migrated table,create a new transfer configwith theOn-demand settings applied.

When you repeat a transfer using an on-demand transfer config,the BigQuery Data Transfer Service attempts to run it as an incremental transfer,but skips the table since the transfer config does not have the correctincremental settings applied. For more information about thedifferent types of transfers, seeOn-demand or incremental transfers.

Issue: Transfer ofCHAR(N) data types adds spaces up to N characters for shorter strings.

Resolution: ConvertCHAR data toVARCHAR and remove the extra spacesat the source. This issue occurs becauseCHAR is a fixed length string,whereasVARCHAR should be used for variable length strings.You can also remove the spaces in BigQuery after the migrationusingtheRTRIM function.The query to use theRTRIM function is similar to the following example:

UPDATE migrated_tableSET migrated_char_column = RTRIM(migrated_char_column)WHERE true;

YouTube transfer issues

The following are common errors encountered whencreating a YouTube transfer.

Error:Import failed - no data was available for import. Please verify that data existence was expected.

Error:No data available for requested date. Please try an earlier run date or verify that data existence was expected.

Resolution: If you have not previously created YouTubereporting jobs,allow YouTube at least 2 days for the BigQuery Data Transfer Serviceto generate the reports on your behalf. No additional action is required.Transfers will fail for the first 2 days and should succeed on day 3. If youhave previously created YouTube reporting jobs, confirm the user creating thetransfers has read access to the reports.

Also, verify the transfer was set up for the correct account. In the OAuthdialog, you must select the channel for which you would like to load data.

Error:No reports for reporting job with namename.

Resolution: This is not an error. It is a warning that indicates no datawas found for the specified report. You can ignore this warning. Futuretransfers will continue to run.

Issue: The resulting tables created by the transfer are incomplete, or the results are unexpected.

Resolution: If you have multiple accounts, you must choose the correctaccount when you receive the YouTube permissions dialog.

Issue: Data doesn't match between YouTube Analytics and BigQuery YouTube Transfers.

Background: BigQuery YouTube transfers useYouTube reporting APIto ingest data directly into a BigQuery dataset. On the otherhand, Youtube Analytics Dashboard pulls data usingYouTube Analytics API.The numbers that YouTube produces in their generated Reporting API should betreated as the final numbers, whereas the numbers visible in the YouTubeAnalytics Dashboard/API should be treated as estimated numbers. Some degree ofdiscrepancy between the two APIs is expected.

Resolution: If the reported numbers are indeed incorrect, thenboth YouTube's system and BigQuery Data Transfer Service YouTube transfer are set up tobackfill missing numbers and make them available in new generated reports forthe backfilled days. Because a BigQuery Data Transfer ServiceYouTube configuration loads all available reports created by YouTubeReporting API, when BigQuerytransfer automatically imports future generatedYouTube Reports, it will also account for the newly generated and updated data,and ingest it into the correct Date partition table.

Note: YouTube data for a given day should be made available in YouTubeAnalytics at most 24-48 hours after the end of that day (sometimes longer).Therefore proceeding generated YouTube Reports andBigQuery Transfers alsofollow this delay.

YouTube permissions issues

For YouTube Content Manager reports, the user setting up the transfer must haveCMS user permissions (at a minimum).CMS user permissions must begranted to each content manager for whom you are creating a transfer.

Quota issues

Error:Quota exceeded: Your project exceeded quota for imports per project.

Resolution: Verify you have not scheduled too many transfers in yourproject. For information on calculating the number of load jobs initiated by atransfer, seeQuotas and limits.