Try BigQuery DataFrames

Use this quickstart to perform the following analysis and machine learning (ML)tasks by using theBigQuery DataFrames API in aBigQuery notebook:

• Create a DataFrame over thebigquery-public-data.ml_datasets.penguinspublic dataset.
• Calculate the average body mass of a penguin.
• Create alinear regression model.
• Create a DataFrame over a subset of the penguin data to use as training data.
• Clean up the training data.
• Set the model parameters.
• Fit the model.
• Score the model.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

1. Verify that billing is enabled for your Google Cloud project.

2. Verify that the BigQuery API is enabled.

   Enable the API

   If you created a new project, the BigQuery API is automatically enabled.

Required permissions

To create and run notebooks, you need the following Identity and Access Management (IAM)roles:

• BigQuery User (roles/bigquery.user)
• Notebook Runtime User (roles/aiplatform.notebookRuntimeUser)
• Code Creator (roles/dataform.codeCreator)

Create a notebook

Follow the instructions inCreate a notebook from the BigQuery editor to create a new notebook.

Try BigQuery DataFrames

Try BigQuery DataFrames by following these steps:

1. Create a new code cell in the notebook.

2. Add the following code to the code cell:

   importbigframes.pandasasbpd# Set BigQuery DataFrames options# Note: The project option is not required in all environments.# On BigQuery Studio, the project ID is automatically detected.bpd.options.bigquery.project=your_gcp_project_id# Use "partial" ordering mode to generate more efficient queries, but the# order of the rows in DataFrames may not be deterministic if you have not# explictly sorted it. Some operations that depend on the order, such as# head() will not function until you explictly order the DataFrame. Set the# ordering mode to "strict" (default) for more pandas compatibility.bpd.options.bigquery.ordering_mode="partial"# Create a DataFrame from a BigQuery tablequery_or_table="bigquery-public-data.ml_datasets.penguins"df=bpd.read_gbq(query_or_table)# Efficiently preview the results using the .peek() method.df.peek()

3. Modify thebpd.options.bigquery.project = your_gcp_project_id line tospecify your Google Cloud project ID. For example,bpd.options.bigquery.project = "myProjectID".

4. Run the code cell.

   The code returns aDataFrame object with data about penguins.

5. Create a new code cell in the notebook and add the following code:

   # Use the DataFrame just as you would a pandas DataFrame, but calculations# happen in the BigQuery query engine instead of the local system.average_body_mass=df["body_mass_g"].mean()print(f"average_body_mass:{average_body_mass}")

6. Run the code cell.

   The code calculates the average body mass of the penguins and prints it to theGoogle Cloud console.

7. Create a new code cell in the notebook and add the following code:

   # Create the Linear Regression modelfrombigframes.ml.linear_modelimportLinearRegression# Filter down to the data we want to analyzeadelie_data=df[df.species=="Adelie Penguin (Pygoscelis adeliae)"]# Drop the columns we don't care aboutadelie_data=adelie_data.drop(columns=["species"])# Drop rows with nulls to get our training datatraining_data=adelie_data.dropna()# Pick feature columns and label columnX=training_data[["island","culmen_length_mm","culmen_depth_mm","flipper_length_mm","sex",]]y=training_data[["body_mass_g"]]model=LinearRegression(fit_intercept=False)model.fit(X,y)model.score(X,y)

8. Run the code cell.

   The code returns the model's evaluation metrics.

Clean up

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as anappspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to theManage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then clickDelete.
3. In the dialog, type the project ID, and then clickShut down to delete the project.

What's next

• Continue learning how touse BigQuery DataFrames.
• Learn how tovisualize graphs using BigQuery DataFrames.
• Learn how touse a BigQuery DataFrames notebook.