Best practices for running an IoT backend on Google Cloud

This document provides security best practices for managing and running anInternet of Things (IoT) backend on Google Cloud. In an IoT solution, anIoT backend connects edge devices to other resources. This document focuses onthe following IoT backends: Message Queuing Telemetry Transport (MQTT) brokerand the IoT platform.

This document is part of a series of documents that provide information aboutIoT architectures on Google Cloud and about migrating fromIoT Core. The other documents in this series include thefollowing:

This document provides best practices for provisioning and managing devicecredentials, authenticating and accessing control edge devices, and letting IoTedge devices access Google Cloud resources.

IoT architecture

An IoT architecture includes services that let you provision and manage devicecredentials, authenticate and access control edge devices, and let edge devicesaccess Google Cloud resources. This document discusses two IoT backendarchitectures: one using MQTT broker and the other using an IoT platform. Themain security differences between these two backends are device identity anddevice management. IoT platforms provide these capabilities as part of theirsystem, whereas MQTT brokers require you to provide these capabilities.

The following diagram describes the IoT architecture.

IoT backend architecture.

The architecture shows the services that are required for the following three processes:

  1. Certificate provisioning, which is the process that you must complete to prepare anedge device for configuration.

  2. Authentication and authorization, which include the authentication scheme that the edge deviceand the MQTT broker or IoT platform use to authenticate with each other.

  3. Connections between edge devices and Google Cloud services, which are tasks that the edge device completes to connect to cloud resources and upload or download data.

This document focuses primarily on security best practices for provisioning and authentication.

The architecture uses a combination of the following services and features:

  • An edge device (such as a medical device) that you deploy at theedges of your environment and that is geographically close to the data that you want to process. Theedge devices connect bi-directionally with your IoT backend, which meansthat they can send messages to and receive messages from the IoT backend.
  • An IoT backend that can be an MQTT broker or an IoT platform.
    • An MQTT broker provides a secure interface for edge devices toconnect using theMQTT protocol.MQTT brokers lack capabilities for device identity and devicemanagement and rely on external systems to provide them.
    • An IoT platform is a cloud application that edge devices connectand communicate with. IoT platforms provide a secure interface for edgedevices to connect using the MQTT protocol. Each IoT platform has itsown security implementation that determines how it authenticates andauthorizes edge devices and how it manages device identities.
  • A central certificate store that hosts the certificates for all edgedevices.
  • Cloud resources that edge devices must access.

Provisioning an edge device

Before an edge device can connect with your backend workloads, you mustprovision a certificate for the edge device. There are two main scenarios thatdecide how you provision the certificate:

  • If your solution is based on commercial, generic devices, you have fullcontrol over the provisioning process after purchasing the device.

  • If you use custom-built devices, the initial provisioning processhappens during the manufacturing of the devices and you must integrate theprovisioning process with your vendors and manufacturers.

In either scenario, you must create device certificates with a chain of trustthat links to a root certificate authority (CA). These certificates authenticatethe device identity and help ensure that updates and modifications done on thedevice are by trusted actors. Use a CA such asCertificate Authority Service to complete the following tasks:

To provision a device certificate, you must complete the following tasks:

  • If your device has a hardware-based, tamper-proof security solution, such asasecure element (SE) or ahardware secure module (HSM) that stores the private keys locally and the private keys are never exposedexternally, do the following:

    1. Generate the public-private key pair using the hardware-based security solution that's supported by your device.
    2. Request a certificate by using acertificate signing request (CSR).

  • If you aren't using a hardware-based security solution to generate apublic-private key pair, use the CA to generate the keys and the certificateinstead. For more information, seeUsing an auto-generated key.The certificate that you download by using this method is already signed.

After you generate and sign a device certificate, you then install the signed devicecertificate on the edge device and store the certificate in a centralcertificate repository, such asSecret Manager.

For more information, seeHow to deploy a secure and reliable public key infrastructure with Google Cloud CA Service (PDF).

For information about other provisioning best practices, seeBest practices for automatically provisioning and configuring edge and bare metal systems and servers.

Three types of certificates are used to help secure an IoT solution:

  • The root CA certificate provides the root for the chain of trust of allthe other certificates in your system. The backend workloads use the rootcertificate to validate the client certificates and the edge devices usethe root certificate to validate the server certificate. You mustdistribute the root certificate to both the IoT backend and to the edgedevices.

  • The intermediate CAs certificates provide a chain of trust that is rootedinto the root CA. You can use intermediate CAs for provisioning, or foroperational needs, such as granting access to intermediate CAs tomanufacturers, or to implement flexible CA management processes.

  • The server certificates are used to secure the endpoints that areexposed by the IoT backend. You have server certificates for the differentencryption algorithms that your endpoints need to support. Servercertificates are linked to the root CA. A secret manager manages and storesboth the private and public portions of the server certificates. You mustconfigure your IoT backend with the server certificates and theircorresponding private keys.

  • The client certificates are used to identify edge devices. Each edgedevice has at least one client certificate, which means that the number ofcertificates that you have increases with the number of edge devices inyour environment. Client certificates are linked to the root CA. You mustdistribute client certificates to your edge devices and to the IoT backend.

Process for generating a device certificate using an HSM or SE

The following diagram shows how a device certificate is provisioned when usingan HSM or SE.

Process for generating a device certificate.

In this diagram, the following steps occur:

  1. The edge device generates the public key pair in the hardware.
  2. You download the public key and create the certificate signing request(CSR) for it.
  3. You send the CSR to the CA to request a certificate.
  4. The CA completes the following actions:
    1. Signs the certificate.
    2. Returns the signed certificate to the provisioner.
  5. The provisioner completes the following actions:
    1. Sends the signed certificate to the edge device.
    2. Stores the signed certificate in the central certificate store.
  6. The edge device stores the certificate in a secure location.

Process for generating a device certificate using the CA

The following diagram shows how a device certificate is provisioned when usinga CA.

Process for generating a device certificate.

In this diagram, the following steps occur:

  1. The provisioner requests that the CA send a signed certificate for thedevice.
  2. The CA completes the following actions:
    1. Generates a public-private key pair and signs the public key.
    2. Returns the device certificate and the private key to the provisioner.
  3. The provisioner completes the following actions:
    1. Sends the certificate and private key to the edge device.
    2. Stores the certificate and private key in the central certificate store.
  4. The edge device stores the certificate and the private key in a securelocation.

If you want to store the private key in a single place (the device), youshould avoid storing the private key in the central secret store. However, ifyou store the private key outside the central secret store and you lose accessto the private key, the device has to go through theprovisioning process again.

Authenticate devices before signing certificates

The process to generate a device certificate (either on the device, or by usinga CA)requires that the device and the CA communicate and authenticate one another.

Without proper authentication, your CA might mistakenly trust a maliciousdevice. For example, an attacker with knowledge about how to reach thecertificate signing infrastructure of your CA might deploy a malicious devicethat asks your CA to sign a certificate. If you don't have device authenticationin place, your CA might sign the certificate that the malicious device presents.If your CA does sign the certificate, the malicious device can then communicatewith your backend as a trusted device.

To help you prevent malicious devices from communicating with your CA, werecommend that you take the following actions:

  • Implement an authentication mechanism for devices that aren't yet trusted.
  • Establish the authenticity of any device requesting authentication.
  • Establish device authenticity before a device asks the CA to generate a new certificate or sign an existing certificate.

Implementing an authentication mechanism at this point of the provisioningprocess is challenging. You cannot rely on device certificates toauthenticate devices because the device doesn't yet have a signed certificatefrom the CA. This lack of a signed certificate can occur for the followingreasons:

  • The device didn't yet generate a certificate.
  • The device didn't yet send a CSR to the CA.
  • The CA didn't yet send the signed certificate back to the device.

One way to solve this problem is to extend your device provisioning process todo the following for each device that you want to authenticate or need toauthenticate:

  1. Generate a provisioning certificate that you use only to authenticate thedevice against the certificate signing infrastructure.
  2. Sign the provisioning certificate with your CA.
  3. Store the signed provisioning certificate in the SE or HSM on the device.
  4. Store the signed provisioning certificate in your Google Cloudbackend.

Before the device is granted access to the certificate signing infrastructure ofyour CA, the device has to present the provisioning certificate. It has topresent the certificate so that you can verify its integrity and authenticity,and determine whether the certificate matches one of the provisioningcertificates stored in your Google Cloud backend. If the verification issuccessful, the device can access the certificate signing infrastructure of yourCA and the certificate provisioning process can continue.

There are differences between a provisioning certificate and a fully-trustedcertificate. A provisioning certificate only grants access to a minimal amountof services and infrastructure. Creating a provisioning certificate lets the CAverify that the device is genuine before considering it fully trusted andissuing a fully-trusted certificate.

An extension to this process is that you can use subordinate CAs that devicemanufacturers have access to, together with your CA, to sign provisioningcertificates. For example, a manufacturer might sign the provisioningcertificates of a device after it completes the manufacturing process for thatdevice. You can then verify these signatures to help you validate that thedevice is genuine.

If a device is compromised before it's provisioned, we recommend that youremove the corresponding provisioning certificate from your Google Cloudbackend, so that the device can't initiate the process to obtain a fully-trustedcertificate because it won't be able to authenticate against your CA.

Best practices for device identity

This section describes the best practices for device identities.

Use an identity provider with MQTT brokers

MQTT brokers authenticate edge devices by using device credentials provided byplugins,databases,andfiles.To manage your device identities in a systematic and scalable manner, use anidentity provider (IdP).The IdP manages the identities and credentials for all devices and acts as theprimary source of truth for device identities.

To keep the device identity updated in the MQTT broker, implement a system-specific integration layer. For more information about managing devicecredentials, seeProvisioning an edge device.

Use the digital identities of the IoT platform as the source of truth

The IoT platform has security features that manage the device identities anddevice credentials, and authenticate and authorize devices trying to access theplatform. These security features help ensure that only authorized devices areallowed to access the IoT platform and help ensure data integrity.

Verify that the device identities managed by the IoT platform represent theprimary source of truth of all the devices the IoT platform manages. Othercomponents in an IoT solution that need device identity information should relyon the security system of the IoT platform. The IoT platform grants accessrights to devices and propagates any security changes throughout the IoTsolution.

Best practices for network connectivity

Securing network connectivity is important for the following reasons:

  • Secure networks help ensure that a device connects to the rightbackend. For example, a secure network can preventDNS spoofing,which is an attack that tries to divert devices to connect to a roguebackend that is controlled by attackers.
  • Secure networks help ensure third parties can't read your data traffic.For example, a secure network can prevent anattacker-in-the-middle attack,where attackers read the traffic between your device and the backend.

UseTransport Layer Security (TLS) to protect network communication between your edge devices and backendworkloads.

Extend TLS withmTLS to implement a mutual authentication scheme that enables both connecting partiesto establish the identity of each other.

For instructions on using TLS, seeStandalone MQTT broker architecture on Google Cloud andIoT platform product architecture on Google Cloud.

Best practices for certificate management for MQTT brokers

This section describes best practices for managing certificates when using MQTTbrokers.

Store certificates centrally

Store and manage server certificates and device certificates in a centrallocation. Specifically, ensure that you have the following controls in place:

  • An inventory of all your devices and their certificates and the serverendpoints and their certificates.
  • Additional information about the certificates such as their validity.
  • The ability to add and remove certificates for devices so that devicescan connect using new certificates.
  • Access rights to your central certificate store, to limit what thedifferent roles in your backend can do with the certificates.

Use a secret storage and management solution such asSecret Manager or HashiCorp Vault. Secret Managerlets you version, update, and invalidate device credentials, and tomanage access policies to your credentials.

For an IoT platform, implement access to the credentials usingSecret Manager API access.

Protect certificates on edge devices

To store certificates and keys on the edge devices, use a localtrusted execution environment or certificate store to protect the credential and block unauthorized accesses.If you need to store secret material on your devices, encrypt that materialusing techniques such asflash encryption,and store it on tamper-proof elements to help prevent unauthorized dataextraction.

Synchronize the central certificate store with the MQTT broker certificate store

MQTT brokers must access client certificates for certificate-basedauthentication, so you must synchronize the certificate stores of MQTT brokerswith the central certificate store. Verify that changes on the centralcertificate store, such as add, update, and delete certificates, aresynchronized with the MQTT broker certificate store. MQTT brokers usecertificate stores such as MySQL, PostgresDB, and Java Key Store. Depending onwhich certificate store your MQTT broker uses, ensure that the followingprocesses exist:

  • A process that monitors for changes in the central certificate storeand notifies the synchronization process.
  • A process that takes changes in the central certificate store andsynchronizes the changes in the central certificate store with thecertificate store used by the MQTT broker.

When you use Secret Manager as your certificate store, youcan useevent notifications as the monitoring process. You can implement the synchronization process as alistener of the event notifications.

Distribute certificates to edge devices securely

When using MQTT brokers, distribute the root certificate and clientcertificates to your edge devices. When you distribute certificates, you mustsecure your communication channels so that the traffic doesn't get intercepted.

The main communication channels for certificate distribution are thefollowing:

  • A direct path from the IoT backend to the edge devices over existingcommunication channels.
  • An indirect path in which edge devices request and download thecertificates.

During certificate distribution, you require the following components:

  • Acertificate store where certificates are centrally managed.
  • Adistribution coordinator which sends the certificates and tracksthe distribution process for each edge device.
  • Anupdate handler on the edge device that receives or downloads thecertificates and stores them on the device.

Distribute certificates during the provisioning processes for edge devices, andwhen you need to rotate certificates.

During the provisioning process, ensure that the provisioner has direct accessto edge devices over encrypted channels such asSSH and uses tools such asSCP.Because the devices are not in operation, you can push certificates directly tothe edge devices.

When rotating certificates, use the MQTT broker as the communication channelbetween the distribution coordinator and the edge devices. Use other channels todownload certificates onto the device. To minimize disruption of the edgedevices in operation, use an indirect certificate distribution path. The processwould consist of the following logical steps:

  1. The distribution coordinator acquires access credentials from thecertificate store.
  2. The distribution coordinator pushes the certificate access credentialsto the edge devices together with additional information, such as thedownload URL.
  3. The on-device update handler receives the access credentials andtemporarily stores the information and acknowledges receipt back.
  4. The update handler coordinates the certificate download when the deviceis not active. The update handler uses the access credentials to downloadcertificates from the credential store.
  5. After the certificates are downloaded, the update handler continues withthe certificate rotation process which is described in thecertificate rotation section.

When you use Secret Manager as the central certificate store,you can generateshort-lived access tokens to grant and restrict access to certificates. For more information, seeDistribute access tokens to devices securely.

To help prevent the certificates from being exposed during transit, encrypt theconnection between your edge devices and the MQTT broker. For more information,seeBest practices for network connectivity.

Rotate certificates automatically

To limit the damage an exposed certificate can cause, generate certificateswith a finite valid period and rotate the certificates before they expire. Forlarge-scale IoT deployments, implement an automatic certificate rotationprocedure to consistently update your devices with new certificates before theold ones expire. Deployed devices without valid certificates means that the devices can stop functioning,which can be costly to fix and negatively affect the overall functionality ofyour IoT solution.

Your edge devices must connect bi-directionally with your MQTT broker to ensurethat they can send messages to the MQTT broker and that they can receivemessages from the MQTT broker.

During certificate rotation, you require the following components:

  • A monitoring process that recurrently scans through your certificateinventory and looks for certificates that are about to expire. Themonitoring process triggers certificate rotation for expiring certificates.
  • A rotation process that initializes and oversees certificate rotation.
  • A device certificate rotation handler on the edge device thatcommunicates with the MQTT broker and executes certificate rotation stepson the device.

To rotate certificates, the IoT solution completes the following steps:

  1. The rotation process sends an initialization message to the edge deviceto start certificate rotation.
  2. The device certificate rotation handler acknowledges the initializationmessage by sending a response back to the rotation job.
  3. The rotation process requests a new certificate from the CA. Thisrequest is similar to thecertificate provisioning request, except that the keys and CSR are sent as MQTT broker messages.
  4. After receiving the new certificate from the CA, the rotation jobdistributes the certificate to the central certificate store and to theedge device. It also synchronizes the certificate to the certificate storeof the MQTT broker.
  5. The device certificate rotation handler stores the new certificate andinitializes a new connection with the MQTT broker using the new certificate.
  6. After the new connection is established, the device certificate rotationhandler sends a completed message to the MQTT broker.
  7. After receiving the completed message, the rotation process invalidatesthe old certificate in the central certificate store.

To help protect the certificates that are being sent during the rotationprocess, use dedicated MQTT topics for certificate rotation. Limit access tothese topics to only the rotation job and the edge device.

To help protect the certificate rotation process from runtime failures, enablepersistence for the changes and progress.

For more information about rotating secrets usingSecret Manager, seeRotation of secrets.

Best practices for certificate management for IoT platforms

If you're using an IoT platform, use the certificate update and distributionmechanisms provided by the platform. For backup purposes, you can regularlyexport the credentials from your IoT platform to a secondary secret storage,such as Secret Manager.

Best practices for authentication with an MQTT broker

During the mutual authentication process, backend workloads verify the identityof edge devices, and edge devices verify the identity of backend workloads.After the backend workloads confirm the identity of the edge device, the backendworkloads authorize the device access to resources.

The following sections provide best practices for authentication methods whenusing MQTT brokers.

Note: Most commercial IoT platform applications support their own set ofauthentication schemes.

Choose the authentication method for MQTT brokers

Different IoT backends support different authentication methods. The commonlyused methods are the following:

  • Username and password authentication, where the edge devicepresents its username and password to verify its identity.
  • Token-based authentication, where encrypted security tokens are usedto verify the edge device's identity.
  • Customized authentication schemes, where you implement a custommechanism to verify the identity of the edge device.

As part of theMQTT standard,MQTT brokers support username and password authentication as the default forMQTT CONNECT packets.

TheMQTT CONNECT packet also contains aClient Identifier field that you can use to uniquely identify the client to the MQTT broker. Edgedevices send theMQTT CONNECT packet to the MQTT broker when they establish aconnection.

Besides the username, password, and client identifier fields in theMQTTCONNECT packet, MQTT 5.0 supportsenhanced authentication that lets you buildchallenge-response authentication flows. MQTT 5.0 allows for multipleAUTH packet exchanges between the edge device and the MQTT broker.

Use password stores with username and password authentication

For username and password authentication, configure the MQTT broker to use apassword store. The password store provides a centralized location for managingpasswords for all the edge devices that connect to the MQTT broker. By default,the username, password, and client identifier fields are optional in the MQTTspecification. Therefore, design your authentication mechanism to verify thatthe username, password, and client identifier fields are present in theMQTTCONNECT packet.

Ensure that the passwords are encrypted at rest and in transit, as follows:

Consider token-based authentication

With token-based authentication, edge devices send a token to the MQTT brokerto authenticate. Devices can generate the token themselves or get the token fromother authentication services. Compared to passwords, tokens are short-lived:tokens are only valid for a period with an explicit expiration date. Alwayscheck for expiration when validating tokens.

JSON Web Tokens (JWT) are a way to implement token-based authentication. Edge devices can generate theJWT and authenticate with the MQTT broker. The JWT is embedded into the MQTTCONNECT packet as the password field.

The advantages of JWT are the following:

  • JWT gives you choices on the encryption algorithm used for signing ofthe token. JWT works well with constrained edge devices, where you can usea less resource-intensive encryption algorithm such as ECC for signing ofthe token.
  • Using public key cryptography, the private key is used only on the edgedevice and never shared with other parties. A private key helps make thismethod more secure than username and password authentication, where thecredentials are sent over the connection and requires encryption of the data.

Consider custom authentication schemes

Some MQTT brokers support different authentication mechanisms and protocols.For example, if your MQTT broker supports customized authentication schemes, youcan configure it to support the following:

Best practices for device access control and authorization

Because of the publisher and subscriber communication pattern of the MQTTprotocol, device access control is defined using MQTT topics. MQTT topicscontrol how a device can communicate with your IoT backend. Each IoT backend hasdifferent implementations for access control and authorization, so refer to yourIoT backend documentation for options on how to set up MQTT topics.

Use single-purpose service accounts to access Google Cloud resources

Access to Google Cloud resources are managed by IAM allow policies that bind the resource access allowance with aset of principals. Typical principals are user accounts, service accounts, andgroups.Service accounts are typically used by an application or compute workload to make authorized APIcalls for cloud resources. Service accounts let IoT edge devices access cloudresources.

Because the device identity is managed by the IoT backend, you must map anidentity between the IoT backend and IAM so that the edge device can accessGoogle Cloud resources.

If you're managing a large set of devices, thelimit on the number of service accounts for each Google Cloud project makes it infeasibleto have direct one-to-one mapping between device and service account.

Instead, create service accounts that are linked to the cloud resources thatyour IoT solution needs to access, as described increating single-purpose service accounts.For example, create a unique service account for each of the following usecases:

  • Downloading software update packages
  • Uploading large media files
  • Ingesting data from a latency stream

To implementleast privilege,ensure that each service account only has sufficient access rights to supportits use case. For example, for a service account that is used to downloadsoftware packages, only grant read access to the Cloud Storage bucket.

Distribute access tokens to devices securely

Usually, your edge devices communicate with your IoT platform using MQTT.However, for specific use cases, your devices might require direct access toGoogle Cloud resources. For example, consider the following:

  • To download content, an edge device requires read-only access to aCloud Storage bucket during the download process only.
  • To upload data to a Cloud Storage bucket, an edge devicerequires write access to the bucket.

For these use cases, useworkload identity federation,where access to Google Cloud resources is granted throughaccess tokens.Workload identity federation eliminates the need of provisioning anycloud-specific credentials on the edge devices and access distribution is donedynamically based on demand.

To distribute access tokens for cloud resources to your devices, configureworkload identity federation between your device identity provider andGoogle Cloud. To support workload identity federation, ensurethat your IoT backend meets theworkload identity federation requirements and follows thesecurity best practices that match your use cases.

To access Google Cloud resources using workload identity federation, youredge devices must implement theOAuth 2.0 Token Exchange workflow, which involves the following steps:

  • The device calls theSecurity Token Service and provides its own device credentials.
  • The Security Token Service verifies the identity of the edge device byvalidating the credentials that the edge device provided with the deviceidentity provider.
  • If the identity verification is successful, the Security Token Servicereturns an access token back to the edge device.
  • The edge device uses that token to impersonate thesingle-purpose service account and obtains ashort-lived OAuth 2.0 access token.
  • The device uses the short-lived OAuth 2.0 access token to authenticatewith Google Cloud APIs and get access to the required cloud resources.

To restrict the access of the short-lived access token to specific buckets andobjects in Cloud Storage, useCredential Access Boundaries.Credential Access Boundaries lets you limit the access of the short-livedcredential and minimize the number of resources that are exposed in yourCloud Storage buckets when an access token gets compromised.

Workload identity federation is a scalable way of securely distributing cloudaccess to edge devices. For more information about authentication, seeAuthentication at Google.

Monitor and audit access to cloud resources

Enable Cloud Audit Logs to create log entries when your edge devices access cloudresources through authenticated API requests. Cloud Audit Logs lets youmonitor critical actions that are done by your edge devices onGoogle Cloud. In addition, Cloud Audit Logs creates the audit traces andlogs that you need to investigate any issues. For more information, seeImpersonating a service account to access Google Cloud.

What's next

Contributors

Authors:

  • Charlie Wang | Cloud Solutions Architect
  • Marco Ferrari | Cloud Solutions Architect

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2024-12-06 UTC.