Run a query
This document shows you how to run a query in BigQuery and understandhow much data the query will process before execution by performing adry run.
Types of queries
You canquery BigQuery databy using one of the following query job types:
Interactive query jobs. Bydefault, BigQuery runs queries as interactive query jobs, whichare intended to start executing as quickly as possible.
Batch query jobs. Batch querieshave lower priority than interactive queries. When a project or reservationis using all of its available compute resources, batch queries are morelikely to be queued and remain in the queue. After a batch query startsrunning, the batch query runs the same as an interactive query. For moreinformation, seequery queues.
Continuous query jobs(Preview).With these jobs, the query runs continuously, letting you analyzeincoming data in BigQuery in real time and then write theresults to a BigQuery table, or export the results toBigtable or Pub/Sub. You can use this capability toperform time sensitive tasks, such as creating and immediately acting oninsights, applying real time machine learning (ML) inference, andbuilding event-driven data pipelines.
You can run query jobs by using the following methods:
- Compose and run a query in theGoogle Cloud console.
- Run the
bq query
command in thebq command-line tool. - Programmatically call the
jobs.query
orjobs.insert
method in the BigQueryREST API. - Use the BigQueryclient libraries.
BigQuery saves query results to either atemporary table (default) or permanent table.When you specify a permanent table as the destination table for the results, youcan choose whether to append or overwrite an existing table, or create a newtable with a unique name.
Note: If you query data from a project to data stored in a different project,the querying project is billed for the query job while the project storing thedata is billed for the amount of data stored in BigQuery.Required roles
To get the permissions that you need to run a query job, ask your administrator to grant you the following IAM roles:
- BigQuery Job User (
roles/bigquery.jobUser
) on the project. - BigQuery Data Viewer (
roles/bigquery.dataViewer
) on all tables and views that your query references. To query views, you also need this role on all underlying tables and views. If you're usingauthorized views orauthorized datasets, you don't need access to the underlying source data.
For more information about granting roles, seeManage access to projects, folders, and organizations.
These predefined roles contain the permissions required to run a query job. To see the exact permissions that are required, expand theRequired permissions section:
Required permissions
The following permissions are required to run a query job:
bigquery.jobs.create
on the project from which the query is being run, regardless of where the data is stored.bigquery.tables.getData
on all tables and views that your query references. To query views, you also need this permission on all underlying tables and views. If you're usingauthorized views orauthorized datasets, you don't need access to the underlying source data.
You might also be able to get these permissions withcustom roles or otherpredefined roles.
Troubleshooting
Access Denied: Project [project_id]: User does not have bigquery.jobs.createpermission in project [project_id].
This error occurs when a principal lacks permission to create a queryjobs in the project.
Resolution: An administrator must grant you thebigquery.jobs.create
permission on the project you are querying. This permission is required inaddition to any permission required to access the queried data.
For more information about BigQuery permissions, seeAccess control with IAM.
Run an interactive query
To run an interactive query, select one of the following options:
Console
Go to theBigQuery page.
Click
SQL query.In the query editor, enter a valid GoogleSQL query.
For example, query theBigQuery public dataset
usa_names
to determine the most common names in the United States between theyears 1910 and 2013:SELECTname,gender,SUM(number)AStotalFROM`bigquery-public-data.usa_names.usa_1910_2013`GROUPBYname,genderORDERBYtotalDESCLIMIT10;
Optional: To select additionalquery settings, click More, and then clickQuery settings.
Click
Run.If you don't specify a destination table, the query job writes theoutput to a temporary (cache) table.
You can now explore the query results in theResults tab of theQuery results pane.
Optional: To sort the query results by column, click
Open sort menunext to the column name and select a sort order. If the estimated bytesprocessed for the sort is more than zero, then the number of bytes isdisplayed at the top of the menu.Optional: To see visualization of your query results, go to theChart tab. You can zoom in or zoom out of the chart, download thechart as a PNG file, or toggle the legend visibility.
In theChart configuration pane, you can change the chart type(line, bar, or scatter) and configure the measures and dimensions of thechart. Fields in this pane are prefilled with the initial configurationinferred from the destination table schema of the query. Theconfiguration is preserved between following query runs in the samequery editor. Dimensions support
INT64
,FLOAT64
,NUMERIC
,BIGNUMERIC
,TIMESTAMP
,DATE
,DATETIME
,TIME
, andSTRING
data types, while measures supportINT64
,FLOAT64
,NUMERIC
, andBIGNUMERIC
data types.Optional: In theJSON tab, you can explore the query results in theJSON format, where the key is the column name and the value is theresult for that column.
bq
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Use the
bq query
command.In the following example, the--use_legacy_sql=false
flag lets you useGoogleSQL syntax.bqquery\ --use_legacy_sql=false\ 'QUERY'
ReplaceQUERY with a valid GoogleSQL query. Forexample, query theBigQuery public dataset
usa_names
to determine the most common names in the United States between the years1910 and 2013:bqquery\--use_legacy_sql=false\'SELECT name, gender, SUM(number) AS total FROM `bigquery-public-data.usa_names.usa_1910_2013` GROUP BY name, gender ORDER BY total DESC LIMIT 10;'
The query job writes the output to a temporary (cache) table.
Optionally, you can specify the destination table andlocationfor the query results. To write the results to an existing table, includethe appropriate flag to append (
--append_table=true
) or overwrite(--replace=true
) the table.bqquery\ --location=LOCATION\ --destination_table=TABLE\ --use_legacy_sql=false\ 'QUERY'
Replace the following:
LOCATION: the region or multi-region for the destinationtable—for example,
US
In this example, the
usa_names
dataset is stored in the USmulti-region location. If you specify a destination table for thisquery, the dataset that contains the destination table must also be inthe US multi-region. You cannot query a dataset in one location andwrite the results to a table in another location.You can set a default value for the location using the.bigqueryrc file.
TABLE: a name for the destination table—for example,
myDataset.myTable
If the destination table is a new table, then BigQuerycreates the table when you run your query. However, you must specifyan existing dataset.
If the table isn't in your current project, then add theGoogle Cloud project ID using the format
PROJECT_ID:DATASET.TABLE
—for example,myProject:myDataset.myTable
. If--destination_table
is unspecified,a query job is generated that writes the output to a temporary table.
API
To run a query using the API,insert a new joband populate thequery
job configuration property. Optionally specify yourlocation in thelocation
property in thejobReference
section of thejob resource.
Poll for results by callinggetQueryResults
.Poll untiljobComplete
equalstrue
. Check for errors and warnings in theerrors
list.
C#
Before trying this sample, follow theC# setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryC# API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
usingGoogle.Cloud.BigQuery.V2;usingSystem;publicclassBigQueryQuery{publicvoidQuery(stringprojectId="your-project-id"){BigQueryClientclient=BigQueryClient.Create(projectId);stringquery=@" SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` WHERE state = 'TX' LIMIT 100";BigQueryJobjob=client.CreateQueryJob(sql:query,parameters:null,options:newQueryOptions{UseQueryCache=false});// Wait for the job to complete.job=job.PollUntilCompleted().ThrowOnAnyError();// Display the resultsforeach(BigQueryRowrowinclient.GetQueryResults(job.Reference)){Console.WriteLine($"{row["name"]}");}}}
Go
Before trying this sample, follow theGo setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryGo API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
import("context""fmt""io""cloud.google.com/go/bigquery""google.golang.org/api/iterator")// queryBasic demonstrates issuing a query and reading results.funcqueryBasic(wio.Writer,projectIDstring)error{// projectID := "my-project-id"ctx:=context.Background()client,err:=bigquery.NewClient(ctx,projectID)iferr!=nil{returnfmt.Errorf("bigquery.NewClient: %v",err)}deferclient.Close()q:=client.Query("SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` "+"WHERE state = \"TX\" "+"LIMIT 100")// Location must match that of the dataset(s) referenced in the query.q.Location="US"// Run the query and print results when the query job is completed.job,err:=q.Run(ctx)iferr!=nil{returnerr}status,err:=job.Wait(ctx)iferr!=nil{returnerr}iferr:=status.Err();err!=nil{returnerr}it,err:=job.Read(ctx)for{varrow[]bigquery.Valueerr:=it.Next(&row)iferr==iterator.Done{break}iferr!=nil{returnerr}fmt.Fprintln(w,row)}returnnil}
Java
Before trying this sample, follow theJava setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryJava API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
importcom.google.cloud.bigquery.BigQuery;importcom.google.cloud.bigquery.BigQueryException;importcom.google.cloud.bigquery.BigQueryOptions;importcom.google.cloud.bigquery.QueryJobConfiguration;importcom.google.cloud.bigquery.TableResult;publicclassSimpleQuery{publicstaticvoidrunSimpleQuery(){// TODO(developer): Replace this query before running the sample.Stringquery="SELECT corpus FROM `bigquery-public-data.samples.shakespeare` GROUP BY corpus;";simpleQuery(query);}publicstaticvoidsimpleQuery(Stringquery){try{// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests.BigQuerybigquery=BigQueryOptions.getDefaultInstance().getService();// Create the query job.QueryJobConfigurationqueryConfig=QueryJobConfiguration.newBuilder(query).build();// Execute the query.TableResultresult=bigquery.query(queryConfig);// Print the results.result.iterateAll().forEach(rows->rows.forEach(row->System.out.println(row.getValue())));System.out.println("Query ran successfully");}catch(BigQueryException|InterruptedExceptione){System.out.println("Query did not run \n"+e.toString());}}}
To run a query with a proxy, seeConfiguring a proxy.
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
// Import the Google Cloud client library using default credentialsconst{BigQuery}=require('@google-cloud/bigquery');constbigquery=newBigQuery();asyncfunctionquery(){// Queries the U.S. given names dataset for the state of Texas.constquery=`SELECT name FROM \`bigquery-public-data.usa_names.usa_1910_2013\` WHERE state = 'TX' LIMIT 100`;// For all options, see https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/queryconstoptions={query:query,// Location must match that of the dataset(s) referenced in the query.location:'US',};// Run the query as a jobconst[job]=awaitbigquery.createQueryJob(options);console.log(`Job${job.id} started.`);// Wait for the query to finishconst[rows]=awaitjob.getQueryResults();// Print the resultsconsole.log('Rows:');rows.forEach(row=>console.log(row));}
PHP
Before trying this sample, follow thePHP setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPHP API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
use Google\Cloud\BigQuery\BigQueryClient;use Google\Cloud\Core\ExponentialBackoff;/** Uncomment and populate these variables in your code */// $projectId = 'The Google project ID';// $query = 'SELECT id, view_count FROM `bigquery-public-data.stackoverflow.posts_questions`';$bigQuery = new BigQueryClient([ 'projectId' => $projectId,]);$jobConfig = $bigQuery->query($query);$job = $bigQuery->startQuery($jobConfig);$backoff = new ExponentialBackoff(10);$backoff->execute(function () use ($job) { print('Waiting for job to complete' . PHP_EOL); $job->reload(); if (!$job->isComplete()) { throw new Exception('Job has not yet completed', 500); }});$queryResults = $job->queryResults();$i = 0;foreach ($queryResults as $row) { printf('--- Row %s ---' . PHP_EOL, ++$i); foreach ($row as $column => $value) { printf('%s: %s' . PHP_EOL, $column, json_encode($value)); }}printf('Found %s row(s)' . PHP_EOL, $i);
Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.cloudimportbigquery# Construct a BigQuery client object.client=bigquery.Client()query=""" SELECT name, SUM(number) as total_people FROM `bigquery-public-data.usa_names.usa_1910_2013` WHERE state = 'TX' GROUP BY name, state ORDER BY total_people DESC LIMIT 20"""rows=client.query_and_wait(query)# Make an API request.print("The query data:")forrowinrows:# Row values can be accessed by field name or index.print("name={}, count={}".format(row[0],row["total_people"]))
Ruby
Before trying this sample, follow theRuby setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryRuby API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
require"google/cloud/bigquery"defquerybigquery=Google::Cloud::Bigquery.newsql="SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` "\"WHERE state = 'TX' "\"LIMIT 100"# Location must match that of the dataset(s) referenced in the query.results=bigquery.querysqldo|config|config.location="US"endresults.eachdo|row|putsrow.inspectendend
Run a batch query
To run a batch query, select one of the following options:
Console
Go to theBigQuery page.
Click
SQL query.In the query editor, enter a valid GoogleSQL query.
For example, query theBigQuery public dataset
usa_names
to determine the most common names in the United States between theyears 1910 and 2013:SELECTname,gender,SUM(number)AStotalFROM`bigquery-public-data.usa_names.usa_1910_2013`GROUPBYname,genderORDERBYtotalDESCLIMIT10;
Click
More, and thenclickQuery settings.In theResource management section, selectBatch.
Optional: Adjust yourquery settings.
ClickSave.
Click
Run.If you don't specify a destination table, the query job writes theoutput to a temporary (cache) table.
bq
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Use the
bq query
commandand specify the--batch
flag. In the following example, the--use_legacy_sql=false
flag lets you use GoogleSQL syntax.bqquery\ --batch\ --use_legacy_sql=false\ 'QUERY'
ReplaceQUERY with a valid GoogleSQL query. Forexample, query theBigQuery public dataset
usa_names
to determine the most common names in the United States between the years1910 and 2013:bqquery\--batch\--use_legacy_sql=false\'SELECT name, gender, SUM(number) AS total FROM `bigquery-public-data.usa_names.usa_1910_2013` GROUP BY name, gender ORDER BY total DESC LIMIT 10;'
The query job writes the output to a temporary (cache) table.
Optionally, you can specify the destination table andlocationfor the query results. To write the results to an existing table, includethe appropriate flag to append (
--append_table=true
) or overwrite(--replace=true
) the table.bqquery\ --batch\ --location=LOCATION\ --destination_table=TABLE\ --use_legacy_sql=false\ 'QUERY'
Replace the following:
LOCATION: the region or multi-region for the destinationtable—for example,
US
In this example, the
usa_names
dataset is stored in the USmulti-region location. If you specify a destination table for thisquery, the dataset that contains the destination table must also be inthe US multi-region. You cannot query a dataset in one location andwrite the results to a table in another location.You can set a default value for the location using the.bigqueryrc file.
TABLE: a name for the destination table—for example,
myDataset.myTable
If the destination table is a new table, then BigQuerycreates the table when you run your query. However, you must specifyan existing dataset.
If the table isn't in your current project, then add theGoogle Cloud project ID using the format
PROJECT_ID:DATASET.TABLE
—for example,myProject:myDataset.myTable
. If--destination_table
is unspecified,a query job is generated that writes the output to a temporary table.
API
To run a query using the API,insert a new joband populate thequery
job configuration property. Optionally specify yourlocation in thelocation
property in thejobReference
section of thejob resource.
When you populate the query job properties, include theconfiguration.query.priority
property and set the value toBATCH
.
Poll for results by callinggetQueryResults
.Poll untiljobComplete
equalstrue
. Check for errors and warnings in theerrors
list.
Go
Before trying this sample, follow theGo setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryGo API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
import("context""fmt""io""time""cloud.google.com/go/bigquery")// queryBatch demonstrates issuing a query job using batch priority.funcqueryBatch(wio.Writer,projectID,dstDatasetID,dstTableIDstring)error{// projectID := "my-project-id"// dstDatasetID := "mydataset"// dstTableID := "mytable"ctx:=context.Background()client,err:=bigquery.NewClient(ctx,projectID)iferr!=nil{returnfmt.Errorf("bigquery.NewClient: %v",err)}deferclient.Close()// Build an aggregate table.q:=client.Query(`SELECT corpus, SUM(word_count) as total_words, COUNT(1) as unique_wordsFROM `+"`bigquery-public-data.samples.shakespeare`"+`GROUP BY corpus;`)q.Priority=bigquery.BatchPriorityq.QueryConfig.Dst=client.Dataset(dstDatasetID).Table(dstTableID)// Start the job.job,err:=q.Run(ctx)iferr!=nil{returnerr}// Job is started and will progress without interaction.// To simulate other work being done, sleep a few seconds.time.Sleep(5*time.Second)status,err:=job.Status(ctx)iferr!=nil{returnerr}state:="Unknown"switchstatus.State{casebigquery.Pending:state="Pending"casebigquery.Running:state="Running"casebigquery.Done:state="Done"}// You can continue to monitor job progress until it reaches// the Done state by polling periodically. In this example,// we print the latest status.fmt.Fprintf(w,"Job %s in Location %s currently in state: %s\n",job.ID(),job.Location(),state)returnnil}
Java
To run a batch query,set the queryprioritytoQueryJobConfiguration.Priority.BATCHwhen creating aQueryJobConfiguration.
Before trying this sample, follow theJava setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryJava API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
importcom.google.cloud.bigquery.BigQuery;importcom.google.cloud.bigquery.BigQueryException;importcom.google.cloud.bigquery.BigQueryOptions;importcom.google.cloud.bigquery.QueryJobConfiguration;importcom.google.cloud.bigquery.TableResult;// Sample to query batch in a tablepublicclassQueryBatch{publicstaticvoidrunQueryBatch(){// TODO(developer): Replace these variables before running the sample.StringprojectId="MY_PROJECT_ID";StringdatasetName="MY_DATASET_NAME";StringtableName="MY_TABLE_NAME";Stringquery="SELECT corpus"+" FROM `"+projectId+"."+datasetName+"."+tableName+" GROUP BY corpus;";queryBatch(query);}publicstaticvoidqueryBatch(Stringquery){try{// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests.BigQuerybigquery=BigQueryOptions.getDefaultInstance().getService();QueryJobConfigurationqueryConfig=QueryJobConfiguration.newBuilder(query)// Run at batch priority, which won't count toward concurrent rate limit..setPriority(QueryJobConfiguration.Priority.BATCH).build();TableResultresults=bigquery.query(queryConfig);results.iterateAll().forEach(row->row.forEach(val->System.out.printf("%s,",val.toString())));System.out.println("Query batch performed successfully.");}catch(BigQueryException|InterruptedExceptione){System.out.println("Query batch not performed \n"+e.toString());}}}
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
// Import the Google Cloud client library and create a clientconst{BigQuery}=require('@google-cloud/bigquery');constbigquery=newBigQuery();asyncfunctionqueryBatch(){// Runs a query at batch priority.// Create query job configuration. For all options, see// https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#jobconfigurationqueryconstqueryJobConfig={query:`SELECT corpus FROM \`bigquery-public-data.samples.shakespeare\` LIMIT 10`,useLegacySql:false,priority:'BATCH',};// Create job configuration. For all options, see// https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#jobconfigurationconstjobConfig={// Specify a job configuration to set optional job resource properties.configuration:{query:queryJobConfig,},};// Make API request.const[job]=awaitbigquery.createJob(jobConfig);constjobId=job.metadata.id;conststate=job.metadata.status.state;console.log(`Job${jobId} is currently in state${state}`);}
Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.cloudimportbigquery# Construct a BigQuery client object.client=bigquery.Client()job_config=bigquery.QueryJobConfig(# Run at batch priority, which won't count toward concurrent rate limit.priority=bigquery.QueryPriority.BATCH)sql=""" SELECT corpus FROM `bigquery-public-data.samples.shakespeare` GROUP BY corpus;"""# Start the query, passing in the extra configuration.query_job=client.query(sql,job_config=job_config)# Make an API request.# Check on the progress by getting the job's updated state. Once the state# is `DONE`, the results are ready.query_job=client.get_job(query_job.job_id,location=query_job.location)# Make an API request.print("Job{} is currently in state{}".format(query_job.job_id,query_job.state))
Run a continuous query
Running a continuous query job requires additional configuration. For moreinformation, seeCreate continuous queries.
Query settings
When you run a query, you can specify the following settings:
Adestination table for thequery results.
The priority of the job.
Whether to usecached query results.
The job timeout in milliseconds.
Whether to usesession mode.
The type ofencryption to use.
The maximum number of bytes billed for the query.
Thedialect of SQL to use.
Thelocation in which to run the query. The querymust run in the same location as any tables referenced in the query.
Thereservation to run yourquery in (Preview).
Optional job creation mode
Optional job creation mode (formerly Short query optimized mode) can improve theoverall latency of queries that run for a short duration, such as those fromdashboards or data exploration workloads. This mode executes the query and returns the results inline forSELECT
statements without requiring the use ofjobs.getQueryResults
to fetch the results. Queries using optional job creation mode don't create ajob when executed unless BigQuery determines that a job creationis necessary to complete the query.
To enable optional job creation mode, set thejobCreationMode
field of theQueryRequestinstance toJOB_CREATION_OPTIONAL
in thejobs.query
request body.
When the value of this field is set toJOB_CREATION_OPTIONAL
,BigQuery determines if the query can use the optional jobcreation mode. If so, BigQuery executes the query and returnsall results in therows
field of the response. Since a job isn't created forthis query, BigQuery doesn't return ajobReference
in theresponse body. Instead, it returns aqueryId
field, which you can use to getinsights about the query using theINFORMATION_SCHEMA.JOBS
view. Since nojob is created, there is nojobReference
that can be passed tojobs.get
andjobs.getQueryResults
APIs to lookup these queries.
If BigQuery determines that a job is required to complete thequery, ajobReference
is returned. You can inspect thejob_creation_reason
field inINFORMATION_SCHEMA.JOBS
view to determinethe reason that a job was created for the query. In this case, you should usejobs.getQueryResults
to fetch the results when the query is complete.
When you use theJOB_CREATION_OPTIONAL
value, thejobReference
field mightnot be present in the response. Check if the field exists before accessing it.
WhenJOB_CREATION_OPTIONAL
is specified for multi-statement queries (scripts),BigQuery might optimize the execution process. As part of thisoptimization, BigQuery might determine that it can complete thescript by creating fewer job resources than the number of individual statements,potentially even executing the entire script without creating any job at all.This optimization depends on BigQuery's assessment of the script, and theoptimization might not be applied in every case. The optimization is fullyautomated by the system. No user controls or actions are required.
To run a query using optional job creation mode, select one of the followingoptions:
Console
Go to theBigQuery page.
Click
SQL query.In the query editor, enter a valid GoogleSQL query.
For example, query theBigQuery public dataset
usa_names
to determine the most common names in the United States between theyears 1910 and 2013:SELECTname,gender,SUM(number)AStotalFROM`bigquery-public-data.usa_names.usa_1910_2013`GROUPBYname,genderORDERBYtotalDESCLIMIT10;
Click
More, and thenchoose theOptional job creation query mode. To confirm this choice,clickConfirm.Click
Run.
bq
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Use the
bq query
commandand specify the--job_creation_mode=JOB_CREATION_OPTIONAL
flag. In the following example, the--use_legacy_sql=false
flag lets you use GoogleSQL syntax.bqquery\ --rpc=true\ --use_legacy_sql=false\ --job_creation_mode=JOB_CREATION_OPTIONAL\ --location=LOCATION\ 'QUERY'
ReplaceQUERY with a valid GoogleSQL query, and replaceLOCATION with a valid region where the dataset is located. Forexample, query theBigQuery public dataset
usa_names
to determine the most common names in the United States between the years1910 and 2013:bqquery\--rpc=true\--use_legacy_sql=false\--job_creation_mode=JOB_CREATION_OPTIONAL\--location=us\'SELECT name, gender, SUM(number) AS total FROM `bigquery-public-data.usa_names.usa_1910_2013` GROUP BY name, gender ORDER BY total DESC LIMIT 10;'
The query job returns the output inline in the response.
Note: you may use--apilog=stdout
to log API requests and responses to extract thequeryId
if needed.
API
To run a query in optional job creation mode using the API,run a query synchronouslyand populate theQueryRequest
property. Include thejobCreationMode
property and set its value toJOB_CREATION_OPTIONAL
.
Check the response. IfjobComplete
equalstrue
andjobReference
is empty, read the results from therows
field. You can also get thequeryId
from the response.
IfjobReference
is present, you can checkjobCreationReason
for why a job was created by BigQuery. Poll for results by callinggetQueryResults
.Poll untiljobComplete
equalstrue
. Check for errors and warnings in theerrors
list.
Java
Available version: 2.51.0 and up
Before trying this sample, follow theJava setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryJava API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
importcom.google.cloud.bigquery.BigQuery;importcom.google.cloud.bigquery.BigQueryException;importcom.google.cloud.bigquery.BigQueryOptions;importcom.google.cloud.bigquery.JobId;importcom.google.cloud.bigquery.QueryJobConfiguration;importcom.google.cloud.bigquery.QueryJobConfiguration.JobCreationMode;importcom.google.cloud.bigquery.TableResult;// Sample demonstrating short mode query execution.//// This feature is controlled by setting the defaultJobCreationMode// field in the BigQueryOptions used for the client. JOB_CREATION_OPTIONAL// allows for the execution of queries without creating a job.publicclassQueryJobOptional{publicstaticvoidmain(String[]args){Stringquery="SELECT name, gender, SUM(number) AS total FROM "+"bigquery-public-data.usa_names.usa_1910_2013 GROUP BY "+"name, gender ORDER BY total DESC LIMIT 10";queryJobOptional(query);}publicstaticvoidqueryJobOptional(Stringquery){try{// Initialize client that will be used to send requests. This client only needs// to be created once, and can be reused for multiple requests.BigQueryOptionsoptions=BigQueryOptions.getDefaultInstance();options.setDefaultJobCreationMode(JobCreationMode.JOB_CREATION_OPTIONAL);BigQuerybigquery=options.getService();// Execute the query. The returned TableResult provides access information// about the query execution as well as query results.TableResultresults=bigquery.query(QueryJobConfiguration.of(query));JobIdjobId=results.getJobId();if(jobId!=null){System.out.println("Query was run with job state. Job ID: "+jobId.toString());}else{System.out.println("Query was run in short mode. Query ID: "+results.getQueryId());}// Print the results.results.iterateAll().forEach(row->{System.out.print("name:"+row.get("name").getStringValue());System.out.print(", gender: "+row.get("gender").getStringValue());System.out.print(", total: "+row.get("total").getLongValue());System.out.println();});}catch(BigQueryException|InterruptedExceptione){System.out.println("Query not performed \n"+e.toString());}}}
To run a query with a proxy, seeConfiguring a proxy.
Python
Available version: 3.34.0 and up
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
# This example demonstrates executing a query without requiring an associated# job.fromgoogle.cloudimportbigqueryfromgoogle.cloud.bigquery.enumsimportJobCreationMode# Construct a BigQuery client object, specifying that the library should# avoid creating jobs when possible.client=bigquery.Client(default_job_creation_mode=JobCreationMode.JOB_CREATION_OPTIONAL)query=""" SELECT name, gender, SUM(number) AS total FROM bigquery-public-data.usa_names.usa_1910_2013 GROUP BY name, gender ORDER BY total DESC LIMIT 10"""# Run the query. The returned `rows` iterator can return information about# how the query was executed as well as the result data.rows=client.query_and_wait(query)ifrows.job_idisnotNone:print("Query was run with job state. Job ID:{}".format(rows.job_id))else:print("Query was run without creating a job. Query ID:{}".format(rows.query_id))print("The query data:")forrowinrows:# Row values can be accessed by field name or index.print("name={}, gender={}, total={}".format(row[0],row[1],row["total"]))
Node
Available version: 8.1.0 and up
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
// Demonstrates issuing a query that may be run in short query mode.// Import the Google Cloud client libraryconst{BigQuery}=require('@google-cloud/bigquery');constbigquery=newBigQuery({// default behavior is to create jobs when using the jobs.query APIdefaultJobCreationMode:'JOB_CREATION_REQUIRED',});asyncfunctionqueryJobOptional(){// SQL query to run.constsqlQuery=` SELECT name, gender, SUM(number) AS total FROM bigquery-public-data.usa_names.usa_1910_2013 GROUP BY name, gender ORDER BY total DESC LIMIT 10`;// Run the queryconst[rows,,res]=awaitbigquery.query({query:sqlQuery,// Skip job creation to enable short mode.jobCreationMode:'JOB_CREATION_OPTIONAL',});if(!res.jobReference){console.log(`Query was run in short mode. Query ID:${res.queryId}`);}else{constjobRef=res.jobReference;constqualifiedId=`${jobRef.projectId}.${jobRef.location}.${jobRef.jobId}`;console.log(`Query was run with job state. Job ID:${qualifiedId}, Query ID:${res.queryId}`,);}// Print the resultsconsole.log('Rows:');rows.forEach(row=>console.log(row));}
Go
Available version: 1.69.0 and up
Before trying this sample, follow theGo setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryGo API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
import("context""fmt""io""cloud.google.com/go/bigquery""google.golang.org/api/iterator")// queryJobOptional demonstrates issuing a query that doesn't require a// corresponding job.funcqueryJobOptional(wio.Writer,projectIDstring)error{// projectID := "my-project-id"ctx:=context.Background()client,err:=bigquery.NewClient(ctx,projectID,bigquery.WithDefaultJobCreationMode(bigquery.JobCreationModeOptional),)iferr!=nil{returnfmt.Errorf("bigquery.NewClient: %w",err)}deferclient.Close()q:=client.Query(`SELECT name, gender, SUM(number) AS totalFROMbigquery-public-data.usa_names.usa_1910_2013GROUP BYname, genderORDER BYtotal DESCLIMIT 10`)// Run the query and process the returned row iterator.it,err:=q.Read(ctx)iferr!=nil{returnfmt.Errorf("query.Read(): %w",err)}// The iterator provides information about the query execution.// Queries that were run in short query mode will not have the source job// populated.ifit.SourceJob()==nil{fmt.Fprintf(w,"Query was run in optional job mode. Query ID: %q\n",it.QueryID())}else{j:=it.SourceJob()qualifiedJobID:=fmt.Sprintf("%s:%s.%s",j.ProjectID(),j.Location(),j.ID())fmt.Fprintf(w,"Query was run with job state. Job ID: %q, Query ID: %q\n",qualifiedJobID,it.QueryID())}// Print row data.for{varrow[]bigquery.Valueerr:=it.Next(&row)iferr==iterator.Done{break}iferr!=nil{returnerr}fmt.Fprintln(w,row)}returnnil}
JDBC Driver
Available version: JDBC v1.6.1 and up
Requires settingJobCreationMode=2
in the connection string.
jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;JobCreationMode=2;Location=US;
LogLevel=6;LogPath=log.txt
to the connection string to enableTRACE
level logging and extract troubleshooting information, includingqueryId
, if needed.ODBC Driver
Available version: ODBC v3.0.7.1016 and up
Requires settingJobCreationMode=2
in the.ini
file.
[ODBCDataSources]SampleDSN=SimbaGoogleBigQueryODBCConnector64-bit[SampleDSN]JobCreationMode=2
LogLevel=6
andLogPath=log.txt
to the.ini
file to enable detailed level logging and extract troubleshooting information, includingqueryId
, if needed.Quotas
For information about quotas regarding interactive and batch queries, seeQuery jobs.
Monitor queries
You can get information about queries as they are executing by using thejobs explorer or by querying theINFORMATION_SCHEMA.JOBS_BY_PROJECT
view.
Dry run
A dry run in BigQuery provides the following information:
- estimate of charges inon-demand mode
- validation of your query
- approximate bytes processed by your query incapacity mode
Dry runs don't use query slots, and you are not charged for performing a dry run.You can use the estimate returned by a dry run to calculate query costs inthepricing calculator.
Note: A dry run of a federated query that uses an external data source mightreport a lower bound of 0 bytes of data, even if rows are returned. This isbecause the amount of data processed from the external table can't be determineduntil the actual query completes. Running the federated query still incursa cost for processing this data.Perform a dry run
To perform a dry run, do the following:
Console
Go to the BigQuery page.
Enter your query in the query editor.
If the query is valid, then a check mark automatically appears along with the amount of data that the query will process. If the query is invalid, then an exclamation point appears along with an error message.
bq
Enter a query like the following using the--dry_run
flag.
bqquery\--use_legacy_sql=false\--dry_run\'SELECT COUNTRY, AIRPORT, IATA FROM `project_id`.dataset.airports LIMIT 1000'
For a valid query, the command produces the following response:
Query successfully validated. Assuming the tables are not modified,running this query will process 10918 bytes of data.
API
To perform a dry run by using the API, submit a query job withdryRun
set totrue
in theJobConfigurationtype.
Go
Before trying this sample, follow theGo setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryGo API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
import("context""fmt""io""cloud.google.com/go/bigquery")// queryDryRun demonstrates issuing a dry run query to validate query structure and// provide an estimate of the bytes scanned.funcqueryDryRun(wio.Writer,projectIDstring)error{// projectID := "my-project-id"ctx:=context.Background()client,err:=bigquery.NewClient(ctx,projectID)iferr!=nil{returnfmt.Errorf("bigquery.NewClient: %v",err)}deferclient.Close()q:=client.Query(`SELECTname,COUNT(*) as name_countFROM `+"`bigquery-public-data.usa_names.usa_1910_2013`"+`WHERE state = 'WA'GROUP BY name`)q.DryRun=true// Location must match that of the dataset(s) referenced in the query.q.Location="US"job,err:=q.Run(ctx)iferr!=nil{returnerr}// Dry run is not asynchronous, so get the latest status and statistics.status:=job.LastStatus()iferr:=status.Err();err!=nil{returnerr}fmt.Fprintf(w,"This query will process %d bytes\n",status.Statistics.TotalBytesProcessed)returnnil}
Java
Before trying this sample, follow theJava setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryJava API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
importcom.google.cloud.bigquery.BigQuery;importcom.google.cloud.bigquery.BigQueryException;importcom.google.cloud.bigquery.BigQueryOptions;importcom.google.cloud.bigquery.Job;importcom.google.cloud.bigquery.JobInfo;importcom.google.cloud.bigquery.JobStatistics;importcom.google.cloud.bigquery.QueryJobConfiguration;// Sample to run dry query on the tablepublicclassQueryDryRun{publicstaticvoidrunQueryDryRun(){Stringquery="SELECT name, COUNT(*) as name_count "+"FROM `bigquery-public-data.usa_names.usa_1910_2013` "+"WHERE state = 'WA' "+"GROUP BY name";queryDryRun(query);}publicstaticvoidqueryDryRun(Stringquery){try{// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests.BigQuerybigquery=BigQueryOptions.getDefaultInstance().getService();QueryJobConfigurationqueryConfig=QueryJobConfiguration.newBuilder(query).setDryRun(true).setUseQueryCache(false).build();Jobjob=bigquery.create(JobInfo.of(queryConfig));JobStatistics.QueryStatisticsstatistics=job.getStatistics();System.out.println("Query dry run performed successfully."+statistics.getTotalBytesProcessed());}catch(BigQueryExceptione){System.out.println("Query not performed \n"+e.toString());}}}
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation. To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.// Import the Google Cloud client libraryconst{BigQuery}=require('@google-cloud/bigquery');constbigquery=newBigQuery();asyncfunctionqueryDryRun(){// Runs a dry query of the U.S. given names dataset for the state of Texas.constquery=`SELECT name FROM \`bigquery-public-data.usa_names.usa_1910_2013\` WHERE state = 'TX' LIMIT 100`;// For all options, see https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/queryconstoptions={query:query,// Location must match that of the dataset(s) referenced in the query.location:'US',dryRun:true,};// Run the query as a jobconst[job]=awaitbigquery.createQueryJob(options);// Print the status and statisticsconsole.log('Status:');console.log(job.metadata.status);console.log('\nJob Statistics:');console.log(job.metadata.statistics);}
PHP
Before trying this sample, follow thePHP setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPHP API reference documentation. To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.use Google\Cloud\BigQuery\BigQueryClient;/** Uncomment and populate these variables in your code */// $projectId = 'The Google project ID';// $query = 'SELECT id, view_count FROM `bigquery-public-data.stackoverflow.posts_questions`';// Construct a BigQuery client object.$bigQuery = new BigQueryClient([ 'projectId' => $projectId,]);// Set job configs$jobConfig = $bigQuery->query($query);$jobConfig->useQueryCache(false);$jobConfig->dryRun(true);// Extract query results$queryJob = $bigQuery->startJob($jobConfig);$info = $queryJob->info();printf('This query will process %s bytes' . PHP_EOL, $info['statistics']['totalBytesProcessed']);
Python
Set theQueryJobConfig.dry_runproperty toTrue
.Client.query()always returns a completedQueryJobwhen provided a dry run query configuration.
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.cloudimportbigquery# Construct a BigQuery client object.client=bigquery.Client()job_config=bigquery.QueryJobConfig(dry_run=True,use_query_cache=False)# Start the query, passing in the extra configuration.query_job=client.query(("SELECT name, COUNT(*) as name_count ""FROM `bigquery-public-data.usa_names.usa_1910_2013` ""WHERE state = 'WA' ""GROUP BY name"),job_config=job_config,)# Make an API request.# A dry run query completes immediately.print("This query will process{} bytes.".format(query_job.total_bytes_processed))
What's next
- Learn how tomanage query jobs.
- Learn how toview query history.
- Learn how tosave and share queries.
- Learn aboutquery queues.
- Learn how towrite query results.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-07-02 UTC.