Deployment sizes

Before deploying Manufacturing Data Engine (MDE), you need to selecta deployment size. This page outlines the available sizes and theircharacteristics. These are general recommendations and may not fit every usecase. For production deployments, consider using a custom size tailored to yourspecific needs.

Sizes

MDE provides four different sizes:

• Pilot: For small tests and proof of concepts, the cheapest option. Thisdeployment size omits the following components (which are included in allother sizes):
  • Bigtable
  • Bigtable Dataflow writer
  • Event Change Dataflow transformation
  • Window Dataflow transformation
• Small: For small projects that expect up to 1k messages per second.
• Medium: For medium projects that expect up to 20k messages per second.
• Large: For large projects that expect up to 100k messages per second.

MDE supports higher throughput if neededthrough custom sizes. If you need to handle a higher throughput than theincluded sizes, contact theMDE teamfor recommendations.

Sizes detail

The following table describes sizes details. This configuration is stored in themde_size_details variable in the filesterraform/modules/infrastructure/variables.tf andterraform/modules/deployment/variables.tf and can be overwritten if needed.

	Pilot	Small	Medium	Large
Max messages/second	300	1000	20000	100000
GKE
CIDR pods	/19 pods	/17 pods	/17 pods	/17 pods
CIDR services	/22 services	/22 services	/22 services	/22 services
compute-class	normal	normal	Scale-Out	Scale-Out
message-mapper max replicas	1	5	50	200
configuration manager max replicas	1	5	50	200
metadata-manager max replicas	1	5	50	200
bigquery-sink max replicas	1	5	50	200
federation-api max replicas	1	2	5	10
SQL
machineType	db-custom-1-3840	db-custom-2-7680	db-custom-16-30720	db-custom-32-61440
max_connections flag	500	500	1000	4000
Redis
tier	Basic	Standard	Standard	Standard
memory	1Gb	5Gn	20Gb	40Gb
read replicas	0	1	2	5
Dataflow
GCSWriter machine type	n1-standard-1	n1-standard-2	n1-standard-4	n1-highmem-4
GCSWriter max workers	1	1	5	10
Bigtable Writer machine type	N/A	n1-standard-2	n1-standard-4	n1-standard-4
Bigtable Writer max workers	N/A	1	3	5
GCSReader machine type	N/A	n1-standard-2	n1-standard-4	n1-standard-4
GCSReader max workers	N/A	1	1	2
EventChange machine type	N/A	n1-standard-2	n1-standard-4	n1-highmem-4
EventChange max workers	N/A	1	2	2
Window machine type	N/A	n1-standard-2	n1-standard-4	n1-highmem-4
Window max workers	N/A	1	2	2
Bigtable
max nodes	N/A	1	3	5

Important: Medium and Large sizes use thescale-out compute class for GKE cluster,which usesT2D CPUs. Additionally, these deployments need at least305and1210 vCPUs respectively when running at full capacity. Make sure torequest this quota extension before trying to deploy MDE.

For more information about how to increase CPU quotas, see theCPU quotas documentation.

Limitations and recommendations

We recommend that you create no more than100 types in thePilot andSmall size deployment and no more than500 types in theMedium andLarge sizes.

This is not a hard limit and MDE cansupport a much higher number of types. Ifyou need support for a higher number of types reach out to theMDE team forassistance in making the necessary Terraform modifications.