Multi-region and custom dual region support for Dataproc Metastore is deprecated. You can no longer create new multi-region or custom dual region services. Existing services will continue to function.

Backported HIVE-28121 to version 3.1.2. Hive Metastore now uses direct SQL for transactionally altering table parameters.

Backported HIVE-26882 to version 3.1.2. Hive Metastore now supports performing transactional checks of table parameters before altering a table.

Dataproc Metastore multi-regional services now support the use ofcustomer-managed encryption keys (CMEKs) -- (inpreview).

Dataproc Metastore federation now supports multi-regional Dataproc Metastore services.

Dataproc Metastore supportscustom region configurations. A custom region configuration lets your service run workloads from two separate regions.

Dataproc Metastoremanaged migrations is generally available (GA)

Dataproc Metastoreautoscaling is generally available (GA)

Dataproc Metastore services can now enabledeletion-protection to prevent the accidental removal of new or existing services.

New Dataproc Metastore services configured with Private Service Connect can be connected from subnetworks of any region within the same VPC network.

Existing services configured with Private Service Connect do not inherit this change and continue to only support access from the VPC subnetworks that were specified during service creation.

Dataproc Metastore now supportsautoscaling. Autoscaling automatically increases or decreases the scaling factor required to run your workloads.

Dataproc Metastore now supportsmanaged migrations.

Managed migration is an automated feature that helps you migrate data from a self-managed Hive Metastore to a Dataproc Metastore service, without any sizable down time.

Dataproc Metastore now supportsscheduled backups. Backups can be scheduled to run at user-specified cron job intervals, including running daily, weekly, or monthly.

The Data Catalog Sync feature is generally available (GA). With this launch, Data Catalog also now supports syncing metadata from Dataproc Metastore services using the Spanner Database.

Dataproc Metastore is available in the followingmulti-regional configurations,nam11 andeur5.

Dataproc Metastore is now available in theme-central2 (Dammam) region. For more information, seeDataproc Metastore locations.

Dataproc Metastore now supportsmulti-regional configurations.

Metadata federation support for BigQuery and BigLake is generally available (GA).

Dataproc MetastoregRPC endpoints are generally available (GA).

Administrator interface is generally available (GA). The GA release adds support for altering Hive table properties.

Metadata federation now supports Dataplex lakes as a metadata source (inpreview)

Dataproc Metastore 2 is now Generally Available (GA). Dataproc Metastore 2 provides horizontal scalability through fine grained scaling options. For more information, seeDatproc Metastore versions.

Auxiliary versions is generally available (GA).

TheSpanner database type is generally available (GA).

Dataproc Metastore is available in the following regions: asia-east2 (Hong Kong), europe-central2 (Warsaw), europe-north1 (Finland), and us-west4 (Las Vagas). For more information, seeDataproc Metastore locations.

Dataproc Metastoreadministrator interface is available inpreview.

The administrator interface provides you with a centralized tool to inspect and manage the metadata stored in your Dataproc Metastore service.

Metadata federation now supports BigQuery datasets as a metadata source (inpreview).

Dataproc Metastore is available in the following regions:us-west2 (Los Angeles),us-west3 (Salt Lake City),europe-west4 (Netherlands),europe-west6 (Zürich), andasia-east1 (Taiwan). For more information, seeDataproc Metastore locations.

Note that these services are immediately available through the gcloud CLI and the REST API. Cloud console availability will vary by region over the next few weeks.

Metadata federation is generally available (GA).

Metadata federation lets you access metadata that is stored in multiple Dataproc Metastore instances.

To set up a federation, you create a federation service and then configure multiple Dataproc Metastore instances as your backend metastores. The federation service then exposes a single gRPC endpoint, which you can use to access metadata across all of your metastore instances.

Private Service Connect for Dataproc Metastore is generally available (GA).

Updated Dataproc Metastore auxiliary versions to support the Spanner database type.

gRPC endpoint protocol is available inPreview.

Fixed the issue causing metadata batch sync from Dataproc Metastore to Data Catalog to not work.

Fixed the issue causing Dataproc Metastore service creation to fail with the errorNO_MATCHING_ACCESS_LEVEL due to a known issue wheredns.googleapis.com is in the service perimeter but not in the allowlist.

Fixed the issue causingrequest_count metric spikes due to a bug in the logic of our metrics reporting pipeline.

Added additional mutual exclusion validation for Data Catalog and Spanner-backed services.

Performing import, export, backup, or restore on Spanner-backed services now returns a 4XX error since these operations aren't supported.

Therequest_count metric spikes due to a bug in the logic of our metrics reporting pipeline.

Creating a Dataproc Metastore service results in the errorNO_MATCHING_ACCESS_LEVEL due todns.googleapis.com in the service perimeter but not in the allowlist. To work around this issue, removedns.googleapis.com from the perimeter during API calls.

Data Catalog sync users must requestroles/metastore.metadataViewer to view synced Dataproc Metastore entries in Data Catalog. Theroles/metastore.Admin androles/metastore.Editor no longer support metastore databases and tables permissions.

Dataproc MetastoreAuxiliary versions are available inPreview.

Spanner database type is available inPreview.

Dataproc Metastore is available in the following regions:

• asia-northeast3 (Seoul)
• southamerica-east1 (São Paulo)

For more information, seeLocations.

AnApache Log4j 2 vulnerability that impacted Dataproc clusters has been addressed (seeRecreate and update a cluster, which provides guidance to Dataproc users). Dataproc Metastore users do not need to take any action; the fix applied by Dataproc Metastore is sufficient to address the issue.

Fixed the issue causing Dataproc Metastore service creations in a VPC-SC perimeter to fail due to a known issue that requires Google-managed service accounts to have access to Dataproc Metastore and Cloud Storage APIs.

Dataproc Metastore service creations in a VPC-SC perimeter may fail due to a known issue that requires Google-managed service accounts to have access to Dataproc Metastore and Cloud Storage APIs.

To work around this issue, or if you encounter a VPC-SC issue of typeNO_MATCHING_ACCESS_LEVEL while creating your Dataproc Metastore service,create an access level for your service so it can access the APIs inside the service perimeter.

Private Service Connect with Dataproc Metastore is available inPreview.

Fixed the issue causing Dataproc Metastore service creations with CMEK enabled to fail if a service without CMEK enabled has never been created before in the project.

CMEK integration with Dataproc Metastore is generally available (GA).

For new projects, Dataproc Metastore service creations with CMEK enabled fail if a service without CMEK enabled has never been created before.

To work around this issue, create a service without CMEK enabled first.

The default Dataproc Metastore service creation version is changed to Hive 3.1.2.

Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 1 week on September 6, 2021.

Fixed the issue causing metadata changes introduced through imports and backups to not be reflected in Data Catalog due to broken batch sync.

CMEK integration with Dataproc Metastore is available inPreview.

Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 5 weeks on September 6, 2021.

Dataproc Metastore is available in the following regions:

• asia-southeast1 (Singapore)
• europe-west1 (Belgium)
• northamerica-northeast1 (Montréal)

For more information, seeLocations.

Dataproc Metastorebackups andrestores support cross-service restoration of metadata.

There is an issue with Dataproc Metastore to Data Catalog batch sync. Metadata changes introduced through imports and backups will not be reflected in Data Catalog until batch sync is restored.

Avro basedimports andexports are generally available (GA).

Backing up andrestoring service metadata are generally available (GA).

Dataproc Metastore performs a Hive metadata schema validation whenimporting metadata into a service.

• For SQL dump, it verifies the tables in the SQL dump file.
• For Avro import, it verifies the Avro file names.
• Both approaches ensure that all tables exist in the import source.

If the verification fails, the operation fails withINVALID_ARGUMENT code and an error message describing which table is missing.

The metadata import history is limited to 25. The oldest import is automatically deleted when the 26th import is created.

You no longer need to manually overridemetastore.expression.proxy to usePartitionProxyForMetastore in Hive 3.1.2.

The asynchronous workflows logs have labels that appear in Cloud logging.

Dataproc Metastore is generally available (GA).

Dataproc Metastoreimports andexports support Avro storage format. This feature is available inPreview.

You canenable sync from a Dataproc Metastore service to Data Catalog. This feature is available inPreview.

Dataproc Metastore supportsasynchronous background tasks through theCanary release channel.

Dataproc Metastore supportsbacking up andrestoring service metadata and configuration. This feature is available inPreview.

Fixed an issue where themetastore.googleapis.com/service/health metric didn't show up for some services.

Fixed an issue where specifying a Cloud Storage URI without an object would return an internal error.

Logs query builder doesn't work when selecting location and service ID.

Hive configuration overrides are rejected if either the key or value contains a newline or "<" character.

TheMetadataImport.DatabaseDump.source_database field is deprecated. It will be removed from the v1beta API channel no earlier than August 18, 2021.

Fixed an issue where services would fail to create in projects with project IDs that contain the colon "(:)" character.

You must havestorage.objects.get permission on the Cloud Storage object in order to import metadata from the Cloud Storage file.

You can create Dataproc Metastore services incross-product networks (shared VPC).

Dataproc Metastore suppports the use ofnon-RFC 1918 private IP address ranges in metastore services.

New Cloud Monitoring service metric is available:

• metastore.googleapis.com/service/request_count

You can update the description of metadata imports under a Dataproc Metastore service.

Fixed an issue where Cloud Storage buckets with single character directories would fail request validation.

Fixed an issue in which a service could get stuck in the UPDATING state.

Fixed the Dataproc Metastore Cloud Logging and Monitoring issue.

Dataproc Metastore Cloud Logging and Monitoring is unavailable. The issue will be fixed shortly.

Dataproc Metastore is available inPreview.

Legacy Dataproc Metastore services created during private Preview (prior to December 14, 2020 at 12:00 PM Pacific Standard Time) will be automatically deleted on January 29, 2021.

The Thrift endpoints of legacy services will continue to function normally, but certain pre-existing functionality such as metadata imports will cease to work. Furthermore, new features (including those announced on December 14, 2020) and bugfixes will not be available to legacy services.

To ensure you receive the newest features, patches, and stability, we strongly recommend you recreate legacy Dataproc Metastore services. Since the new metadata export feature is not available for legacy services, if you need help migrating metadata from a legacy service, the Dataproc Metastore team will be happy to assist you with a manual migration.

Please contact dataproc-metastore-support@google.com with any questions or to request help migrating metadata.