Download as PDF, PPTX
Uploaded byToru Makabe
PDF, PPTX2,883 views
Azure Kubernetes Service 2019 ふりかえり
This document provides an overview and summary of key releases and features for Azure Kubernetes Service (AKS) in 2019. It begins with introductions from the author and describes the major AKS releases for 2019, including availability zones, multiple node pools, cluster autoscaler, network policy, and more. It also summarizes major releases for Azure Container Registry and discusses upcoming features on the public roadmap. Other related cloud-native projects from Microsoft are listed as well. The document concludes with the author's perspectives on best practices for AKS at the end of 2019.
Recommended
![[Cloud OnAir] Google Cloud で実現するバックアップ ディザスタリカバリのベストプラクティス 2019年4月25日 放送](/image.pl?url=https%3a%2f%2fcdn.slidesharecdn.com%2fss_thumbnails%2f0425-190425095700-thumbnail.jpg%3fwidth%3d640%26amp%3bheight%3d640%26amp%3bfit%3dbounds&f=jpg&w=240)
More Related Content
What's hot
Similar to Azure Kubernetes Service 2019 ふりかえり
More from Toru Makabe
Recently uploaded
Azure Kubernetes Service 2019 ふりかえり
- 1.Azure Kubernetes Service真壁徹日本マイクロソフト株式会社クラウドソリューションアーキテクト2019/12/42019年を ふりかえる
- 2.自己紹介apiVersion: selfIntroduction/v1name: “真壁徹(まかべ とおる)”company:name: “日本マイクロソフト株式会社”role: “クラウド ソリューションアーキテクト”career:- name: “大和総研”- name: ”HP Enterprise”cert : “CNCF Certified Kubernetes Admin.”
- 3.
- 4.Azure Kubernetes Service(AKS) 主なリリース補足のないものはGAしてますAPI ServerのIPアドレス ホワイトリスト制御APIサーバー 監査ログAzure Monitor for containersVirtual Node in ACIユーザー定義ルートのサポートAzure Arc による管理 (Preview)Kubernetes 1.12、1.13、1.14、1.15(Preview)Availability Zones複数Node PoolCluster Autoscaler & VMSSNetwork PolicyJapan East/West両リージョン対応
- 5.Azure Kubernetes Service(AKS) 主なリリース補足のないものはGAしてますAKS Pod Security Policy(Preview)Azure PolicyとOpen Policy Agentの統合 (Preview)Azure Monitor for containers ライブデータ表示 (Preview)Azure Monitor PrometheusMetric スクレイピング (Preview)Standard Load BalancerサポートApp Gateway Ingress Controller証明書ローテーションEgress lockdown対話型診断Managed Identity 統合 (Preview)Windowsノード (Preview)
- 6.Azure Container Registry(ACR) 主なリリース補足のないものはGAしてますリポジトリスコープ RBACサポート (Preview)監査と診断ログ (Preview)Azure Security Centerによるイメージの脆弱性スキャン(Preview)署名済みイメージのサポートACR Tasks スケジューリングACR in VNet (Preview)Helm Chartリポジトリ (Preview)ACR Tasks Cloud Native Buildpackサポート (Preview)
- 7.
- 8.そのほかリリースマイクロソフトがリード/関与が強い CNCFプロジェクトなどKubernetes ConfidentialcomputingDistributedApplication Runtime(Dapr)Cloud Native Application Bundle(CNAB) with BrigadeKEDA (Kubernetes-based EventDriven Autoscaling)GitHub Actions for deploying toKubernetes serviceService Mesh InterfaceHelm 3
- 9.
- 10.
- 11.Ignite 2019 BreakoutSessionBRK4006 Applying best practices to Azure Kubernetes Service (AKS)AKSを本番で使うなら必見高可用性バックアップ&リストアマルチクラスター & マルチリージョンアップグレードなどなど
- 12.不安定な時はノードのディスク性能を疑うkubeletやtunnnelfrontなど、ノード上にはアプリ以外にも重要コンポーネントが動いているIOPSの高いディスクにすると安定することが とても とても多いPodのResource Limitも重要 暴れん坊を抑え込もう特にメモリ周辺リソースと合わせてInfrastructure as Code化するアップグレードや新機能の検証など、運用しているとクラスターを作る機会は多いKubernetesだけで完結するシステムは稀データストアやネットワーク系サービスなど周辺リソースも合わせて作れるようにするTerraformでもARM Templateでも お好みで俺のベストプラクティス
- 13.おかしいな、という時はGitHubのIssueを検索するサポート窓口ではありません が、PMがIssueを見て対応していますズバリな解決策が見つかることもとはいえ サポート要求はサポート窓口、新機能リクエストはAzureFeedbackへ話題の機能/OSSだからといって あれもこれも使う必要はない運用するなら腹落ちしたもの、ビジネス目標に合うものを選んで使いましょう華やかな 事例の裏には 生存者バイアスがみなぎらないなら その時を待つどんどん進化するので 追従には組織としてのモチベーションが重要です流行ってるっぽいから、誰かにやれって言われたから、では続きませんAKS/Kubernetesの代替方式は多くあります (App Service、Functions、ACI、etc)俺のベストプラクティス
- 14.
- 15.
- 16.Additionalnodes neededPods areinpending stateCluster AutoscalerPod PodNodePod PodNodePod PodAKS clusterClusterAutoscalerAzureNode is grantedPending podsare scheduledThe cluster autoscaler watches for pods that can't be scheduled on nodes because ofresource constraints. The cluster then automatically increases the number of nodes.1. HPA obtains resource metrics andcompares them to user-specifiedthreshold2. HPA evaluates whether user specifiedthreshold is met or not3. HPA increases/decreases the replicasbased on the specified threshold4. The Deployment controller adjuststhe deployment based onincrease/decrease in replicas
- 17.• Elastically provisioncompute capacityin seconds• No infrastructure to manage• Built on open sourced Virtual Kubelettechnology, donated to the Cloud NativeComputing Foundation (CNCF)Serverless Kubernetes using AKS virtual nodesNodePodsNodePodsKubernetescontrol planeAzure ContainerInstances (ACI)PodsVirtual node
- 18.Application Gateway IngressController• Application Gateways as Ingress for AKS• Deployed using Helm• Utilizes pod-AAD for ARM authentication.• Tighter integration with AKS add on support upcoming• Supports URI path based, host based, SSL termination,SSL re-encryption, redirection, custom health probes,draining, cookie affinity• Support for Let’s Encrypt provided TLS certificates• WAF fully supported with custom listener policies• Support for multiple AKS as backend• Support for mixed mode – both AKS and other backendtypes on the same Application Gateway!ApplicationGatewayAzure ARMAzure Key VaultAzure KubernetesServices (AKS)APIserverAGIngressControllerIngressresourcePodsConfigure routing rules
- 19.Accelerate containerized developmentKubernetesand DevOpsbetter togetherDevelop• Native containers and Kubernetes support in IDE• Remote debugging and iteration for multi-containers• Effective code merge• Automatic containerizationDeliver• CI/CD pipeline with automated tasks in a fewclicks• Pre-configured canary deployment strategy• In depth build and delivery process review andintegration testing• Private registry with Helm supportOperate• Out-of-box control plane telemetry, logaggregation, and container health• Declarative resource management• Auto scalingInner loopTestDebugAzureDevSpacesAKS devclusterCI/CD PipelinesGitHub reposAzureContainerRegistryHelm chartContainerimageAKSproductionclusterAzureMonitorScaleTerraformDevelop Deliver OperateBoards
- 20.GitHub Actions forKubernetes on Azure1. Authenticate and login securely to an Azuresubscription2. Set the target AKS cluster3. Create Kubernetes secret objects to managesensitive information4. Connect to the Kubernetes cluster and deploymanifests, etc.Actiondocker-loginActionk8s-create-secretActionaks-set-contextActionk8s-deploy
- 21.Pull Request flowin Dev Spaces1. John is working out of branch “feature-x”locally2. John commits his code and pushes hisbranch to his remote GitHub repo3. John creates a pull request beforemerging the changes into theapplication’s main branch4. GitHub Actions workflow is triggeredupon PR creation; a delta namespace forthe pull request is created and the codeis deployed to the namespaceSourcecode controlmasternamespacefeature-xnamespaceGitHub Actionsworkflow builds anddeploysfeature-xJohnDeveloperLisaReviewerOpen pullrequest, deployfeature-branchPull request merged,master updatedPR namespacecreated, changesdeployedAzure Dev Spaces + AKS cluster5. A team member reviews the changes inthe context of the entire application6. The pull request is approved and aGitHub workflow is triggered to updatethe master namespace with the mergedcode changes
- 22.AKS with RBACStorageSQLDatabaseCosmosDBVNetNodeNodePod PodAAD Pod IdentityKey VaultActiveDirectoryActiveDirectoryIdentityUse familiar tools like AAD for fine-grained identity and access control to Kubernetes resources fromcluster to containers
- 23.1. Kubernetes operatordefines anidentity map for K8s service accountsAAD Pod identity2. Node Managed Identity (NMI)watches for mapping reaction andsyncs to Managed Service Identify(MSI)3. Developer creates a pod witha service account, and pod usesstandard Azure SDK to fetcha token bound to MSI4. Pod uses access token to consumeother Azure services; services validatetokenKubernetesKubernetescontrollerAzure MSIAzureIdentityBindingActiveDirectoryPod IdentityNMI + EMSIPodTokenAzure SQLServerDeveloper<¥>
- 24.1. Cloud architectassigns a deploymentpolicy across cluster(s)2. Developer uses standard KubernetesAPI to deploy to the cluster3. Real-time deployment enforcement(acceptance/denial) provided todeveloper based on policy4. Cloud architect obtains compliancereport for the entire environment andcan drill down to individual pod levelAzure Policy for clusters (OPA Integration)CloudArchitectDeveloperCluster-1Cluster-2 Cluster-3AKSAzurePolicyCluster-3Cluster-2Cluster-1Compliance reports
- 25.2. Node andcluster level security• Automatic security patching nightly• Nodes deployed in private virtual networksubnet w/o public addresses• Network policy to secure communicationpaths between namespaces (and nodes)• Pod Security Policies using Gatekeeper• K8s RBAC and AAD for authentication• Threat protection on nodesAKS with RBACSecurity overview1. Image and container level security• AAD authenticated Container registryaccess• ACR image scanning and content trust forimage validation3. Pod level security• Pod level control using AAD Pod Identity• Pod Security Context4. Workload level security• Azure Role-based Access Control (RBAC)& security policy groups• Secure access to resources & services(e.g. Azure Key Vault) via Pod Identity• Storage Encryption• App Gateway with WAF to protect againstthreats and intrusionsDeveloperAzureContainerRegistryKubernetesAdminAzure Storage SQL Database Cosmos DBInternalUserInternalLoad BalancerExternalUserExternalLoad BalancerAzure VNetNode NodePod PodAAD Pod IdentityIngressControllerEncrypted StorageAzureKey VaultIngressControllerApp GatewayExternalDNSActiveDirectory
- 26.AKS Support inAzure Security Center1. For managed subscriptions, each new AKScluster and node are discovered in ASC2. ASC monitors AKS cluster for securitymisconfigurations and providesactionable recommendations forcompliance with security best practices3. ASC continuously analyzes AKS forpotential threats based on:a. Raw security events such as networkdata and process creationb. Kubernetes log auditAzureSecurityCenterContinuous discovery ofmanaged AKS instancesActionable recommendationsfor security best practicesDetect threats across AKSnodes and clusters usingadvanced analyticsAzure Kubernetes ServiceAKS security configurationAPI ServerMasterWorkersNode1Container runtimeSecurity centerNode2Container runtimeSecurity centerNode3Container runtimeSecurity centerVerified by Security CenterAudit logRaw security events…and reports any threats and maliciousactivity detected (e.g., “API requests to yourcluster from a suspicious IP was detected”)
- 27.Threat protectionAutomated threatdetection and best practices recommendation for Kubernetes clusters using advancedanalytics from Azure Security CenterCluster Cluster ClusterAzureSecurityCenterContinuous discovery ofmanaged AKS instancesActionable recommendationsfor security best practicesDetect threats across AKSnodes and clusters usingadvanced analyticsAzure Kubernetes Service
- 28.Image SecurityDeveloperCI/CDPipelinesAzure ContainerRegistryAzureKubernetesServiceImage scanningFail PassYour private registry, with built-in Helm chart support, only deploys validated images and can beautomatically geo-replicated to the data center close to where your users areVulnerabilityscanningActionablerecommendationsAdmin
- 29.Secure network communicationswith VNET and CNIAKS subnetBackendservices subnetAzure VNet AOn-premisesinfrastructureEnterprisesystemOther peered VNetsVNet peeringAzureExpressRouteAKS cluster SQL Server1. Uses Azure subnet for both yourcontainers and cluster VMs2. Allows for connectivity to existingAzure services in the same VNetAKS VNet integration works seamlesslywith your existing network infrastructure3. Use Express Route to connect to on-premises infrastructure4. Use VNet peering to connect to otherVNetsServiceEndpointAzure SQLPaaS DB5. Connect AKS cluster securely andprivately to other Azure resourcesusing VNet endpoints
- 30.1. A developerauthenticates to the AADtoken issuance endpoint and requestsan access tokenIdentity and access management through AAD and RBACAzure delivers a streamlined identity andaccess management solution with AzureActive Directory (AAD) and AzureKubernetes Services (AKS)2. The AAD token issuance endpointissues the access token3. The access token is used toauthenticate to the secured resource4. Data from the secured resource isreturned to the web applicationAKSAzure ActiveDirectoryTokenTokenDeveloper
- 31.Azure Pipelines buildaudit & enforcement using Azure Policy1. Cloud architect assigns a policy acrossclusters; policy can be set to block non-compliance (deny) or generate non-compliance warnings (audit)2. Developer makes code change that kicksoff a build on Azure Pipelines3. Azure Pipelines evaluates the request forpolicy compliance4. If policy is set to deny, Azure Pipelinesrejects the build attempt if any non-compliance is identified5. If policy is set to audit, a non-complianceevent is logged and the build is allowedto proceedCloudArchitectDeveloperCluster-1 Cluster-2 Cluster-3AKSAzurePolicyCI/CD PipelinesPassFailDeny policy</>Yes NoCompliance check</>
- 32.Azure managementexperiencesAzure PortalAzureCLIAzure SDKHybrid Agentand Services:Azure Arc for Kubernetes - ComponentsAzureIdentityRBACPolicyIndexGroupsEtc.Azure ResourceManagerAzureContainerRegistryKubernetesK8s API serverCluster provisioningCluster upgrade and patchmanagementCluster lifecycle managementCluster monitoringAdministrative accessK8s native toolsCustomer locationsConfig ServiceK8s ConnectServiceSource RepoGitOpsManagerConfig AgentConnect Agent
- 33.k8s cluster1: security.yamlClusterAdmin3:Arcoperators4:KubernetesCluster –Azure Arc2: PolicySecurityAdmin6: configto cluster8: get manifestfrom repo7: Git Url9: apply andenforce rulesAzure Arc for Kubernetes - Workflow5
- 34.Azure Monitor forcontainers AzureMonitorfor containersVisualizationInsightsMonitor &analyzeResponseNative alerting with integrationto issue management andITSM toolsMonitor and analyzeKubernetes and containerdeployment performance,events, health, and logsProvide insights withcluster health rollup viewVisualize overall health andperformance from cluster tocontainers with drilldownsand filtersCloud native experiencefor Azure Monitor withPrometheus integrationAzureKubernetesServiceAzurePipelinesObservabilityObserve live containerlogs and Kubernetesevent log on containerdeployment statusVirtualnodePrometheus1. Get detailed insights about yourworkloads with Azure Monitor3. See graphical insights about clusters2. Filter for details about nodes,controllers, and containers4. Pull events and logs for detailedactivity analysis
- 35.1. Deploy AzureArc for Kubernetesagent2. Azure Arc agent registers cluster withARM3. Cluster operator applies clusterconfiguration via ARM4. Configuration agent picks upconfiguration and syncs statefrom git repo5. Configuration agent informs Azurepolicy of status6. Cluster operator or applicationdeveloper pushes changes via GitHubClusterConnect RPClusteroperatorAzureResourceManagerClusterConfig RPAzurePolicyGitHubConfigagentAzure ArcagentCluster operator/Application devAzure Monitor for containersConfiguration management scenarioKubernetes on-prem
- 36.AKS DiagnosticsAzurebackendtelemetry Node1 Node 2AKSproductionclusterUserAzureportalZero configurationand zero costIntelligent detectors basedon AKS-specific telemetryCluster-specificobservationsRecommended actionsfor troubleshooting<¥>Cluster InsightsCluster Node IssuesNode Issues DetectedNode Insufficient Resources DetectedCreate, Read, Update & Delete OperationsIdentity and Security ManagementAKS DiagnosticsSample diagnostics web portal!!!xAn interactive and intelligent experiencefor self-troubleshooting your app issuesDiagnose and guide you through eachproblem with best practicesrecommendationsIntelligent search capabilities to help youfind right answers fastStraight out-of-the box, no extraconfiguration necessary
- 37.Open-source component jointlybuilt by Microsoft andRedHat• Event-driven container creation & scalingAllows containers to “scale to zero” until an eventcomes in, which will then create the container andprocess the event, resulting in more efficientutilization and reduced costs• Native triggers supportContainers can consume events directly from theevent source, instead of routing events throughHTTP• Can be used in any Kubernetes serviceThis includes in the cloud (e.g., AKS, EKS, GKE, etc.)or on-premises with OpenShift—any Kubernetesworkload that requires scaling by events instead oftraditional CPU or memory scaling can leverage thiscomponent.Kubernetes-based event-driven auto-scaling (KEDA)Kubernetes clusterExternaltrigger sourceKEDAAKS clusterScalerControllerMetrics adapter
- 38.SMI defines aset of APIs that can be implementedby individual mesh providers. Service meshes and toolscan either integrate directly with SMI or an adapter canconsume SMI and drive native mesh APIs.• Standard interface for service mesh on Kubernetes• Basic feature set to address most commonscenarios• Extensible to support new features as they becomewidely availableService Mesh Interface (SMI)Apps Tooling Ecosystem…and moreService Mesh InterfaceRouting Telemetry PolicyKubernetes
- 39.
- 40.Bosch Increases VehicleSafety Using PrecisionGPS Algorithms and Azure Kubernetes ServiceChallenge: Bosch designed a software development kit (SDK) thatcan be used by original equipment manufacturers (OEMs)to embed driving safety information at scale. For such aservice to work commercially, they had to build a real-time data ingestion and processing pipeline capable ofdetecting hazards and notifying drivers within secondsSolution: The solution is deployed as multiple microservicesrunning in containers behind an Azure API Managementgateway. AKS provided the simplicity a serverlessKubernetes experience that provided the elasticprovisioning they wanted without the need to manage theinfrastructure.Outcome: By running their solution, which has been downloaded by12 million users, on Azure and AKS, the average time todetect driving hazards dropped to approximately 60milliseconds.What we like about AKS is the simplifiedKubernetes experience. It's click and deploy,it’s click and scale. It’s infrastructure as codetoo, which is quite cool for us.”— Christian Jeschke, Product Owner, BoschClick icon to learn more“
- 41.Bosch: architecture1. Sensordata is generated and streamed toAzure API Management2. AKS cluster runs microservice that are deployedas containers behind a service mesh; containersare built using a DevOps process and stored inAzure Container Registry3. Ingest service stores data in an Azure CosmosDB and other data storage destinations4. Asynchronously, the map matching servicereceives the data from Kafka Streams on AzureHDInsight5. Data is processed and stored the result inAzure Database for PostgreSQL and maps arecontinuously updated using Azure Databricks6. A web app running in Azure App Service is usedto visualize the resultsVNetSecurityPublic API Key VaultSDKHotspotsWDW ServiceBlobStorageWeb AppsACRAKSServiceKafka Streamson HD InsightsAKSMap matchingData ExplorerClustersCosmosDBCache forRedisPostgreSQLServerDatabricksmVISE
- 42.Power grid operatoruses containerized softwareto promote smart utility initiatives for 1.5M peopleChallenge: Legacy systems for reading meter data needed greatercapacity to process large volumes of IoT data—butimplementing the necessary system enhancements wasdifficult and expensiveSolution: Hafslund chose to develop its own software forprocessing meter data. The company used MicrosoftAzure as its cloud platform, AKS to manage softwarecontainers, and Azure Monitor for containers to optimizecontainer performance.Outcome: Halfslund now has a standard way to create, monitor,scale, and manage applications, which means it canrespond to customer needs faster.We wanted a platform to speed developmentand testing but do it safely, without losingcontrol over security and performance. That’swhy Azure and AKS are the perfect fit for us.”— Ståle Heitmann, Chief Technology OfficerHafslund NettClick icon to learn more“
- 43.Hafslund Nett: architecture1.Azure Pipelines automates containerimage build, push and release to AzureKubernetes service, triggered by sourcecode updates.2. Azure Kubernetes Services provides thealways-on service for meter reading andconnects with Azure managed databasesto process the massive amounts of datathe IoT devices generate3. Azure API Management serves as thesecure gateway that helps connect todata and services anywhere.4. Azure network and Active Directoryprovides fine-grain controls for externaland inter-service communication.5. Azure Monitor provides a single pane ofglass for cluster-to-container monitoring.…TerraformInfrastructureAKS…Namespace…Namespace NamespaceHalsfundNettCSS StylesExpressRoutesVirtualNetworkTableStorageGitHubVMActiveDirectoryKey Vault…Application InsightsLogAnalyticsCosmosDBSQLServerAzureSearchContainerMonitorOn-premservicesVM VM VMLoadBalancerInternalACRDevOpsLoadBalancerExternalAPIMgmt.
- 44.DNV GL scalesup machine learning using AzureKubernetes ServiceChallenge: Initially, the group trained machine learning modelslocally and deployed each application to Azure VirtualMachines. This process took up to 2 weeks and consumedmore Azure resources than needed.Solution: DNV GL created a service using that builds and deployseach machine learning application as a container on AKS.They’re able to use the Kubernetes Cluster Autoscaler toadd resources on demand as the need for more computepower arises.Outcome: Data scientists and developers at DNV GL can now delivermore solutions to their internal and external customerswith more speed, for less money, and with a more elasticsoftware stack. Now the data scientists and engineers atDNV GL can focus on developing new, predictive solutionsand providing real business value.Click icon to learn moreWe decided to address the friction areasof our internal company deployment,management, and operations, and afterevaluating commercial offerings, we choseto develop ML Factory based on Azureservices.”— Kristian Ramsrud, Machine Learning groupDNV GL Maritime“
- 45.DNV GL: architecture1.Data scientists create their machine learningapplications as containers using the ML Factorydevelopment tools2. ML Apps are built using automatically using AzureContainer Registry Tasks and are deployed to AzureKubernetes Service3. Realtime logs can be streamed directly fordebugging purposes. Azure Log Analytics alsoprovides access to historical logs within definedretention periods4. As the data flows through the platform, multiplefunctions hosted in Azure Functions work togetherto fire alerts or trigger actions, triggered by signalsfrom Azure Event Grid5. Published applications are automatically added tothe company’s corporate API Management gatewayand the internal API catalogML developmentand monitoringSupportcomponentsML FactoryEventGridFunctionAppsML FactoryDeveloper toolsActiveDirectoryBlobStorageAPIManagementKeyVaultML FactoryWeb portalAKSSQLServerStorageAccountsAppServiceAPIGatewayConsumingapplicationsACR
- 46.Click icon tolearn moreMaersk uses AKS for a customer service processto elevate NSAT, an industry-wide challengeNeeds: Get near-real-time data to provide better customerserviceCollect data for future Machine Learning driven featuresChallenges: Compute & memory intensive featuresData integration difficultiesLimited organisational experience in Cloud & KubernetesRequirements: Spend less time on container software managementAutomation and continuous deliveryFull visibility to application, container and infrastructureFine grained security and access controlOutcomes: Reduced environment provisioning time from 1+ weeksto 2.5 hoursAKS and CaaS can potentially save 33% on run costUsing Kubernetes on Azure satisfies ourobjectives for efficient software development.It aligns well with our digital plans and ourchoice of open-source solutions for specificprogramming languages.”“— Rasmus Hald, Head of Cloud Architecture,A.P. Moller - Maersk
- 47.1. Azure Pipelinesfor automationand CI/CD pipelines; addingTerraform for further automation2. Key Vault to secure secrets andfor persistent configurationstore3. Azure Monitor for containers toprovide better logging,troubleshooting, with no directcontainer access4. RBAC control for fine grainedKubernetes resources accesscontrolFirewallApp GatewayAKS w/RBACAzureMonitorAppInsightsSQLDatabaseCosmosDBPerformanceDocument DBKey VaultVaultEvent HubBatch processingEvent SimulationDataFactoryDataManagementGatewayOn-premisesdatabaseExpressRouteService BusInternal QueuingSQLDatabaseAzurePipelinesMaersk: architecture
- 48.© Copyright MicrosoftCorporation. All rights reserved.