Tutorial: Create a four-stage pipeline
Now that you've created your first pipeline inTutorial: Create a simple pipeline (S3 bucket) orTutorial: Create a simple pipeline (CodeCommit repository), you can start creating more complex pipelines. This tutorial will walk you through the creation of a four-stage pipeline that uses a GitHub repository for your source, a Jenkins build server to build the project, and a CodeDeploy application to deploy the built code to a staging server. The following diagram shows the initial three-stage pipeline.
After the pipeline is created, you will edit it to add a stage with a test action to test the code, also using Jenkins.
Before you can create this pipeline, you must configure the required resources. For example, if you want to use a GitHub repository for your source code, you must create the repository before you can add it to a pipeline. As part of setting up, this tutorial walks you through setting up Jenkins on an EC2 instance for demonstration purposes.
Many of the actions you add to your pipeline in this procedure involve AWS resources that you need to create before you create the pipeline. AWS resources for your source actions must always be created in the same AWS Region where you create your pipeline. For example, if you create your pipeline in the US East (Ohio) Region, your CodeCommit repository must be in the US East (Ohio) Region.
You can add cross-region actions when you create your pipeline. AWS resources for cross-region actions must be in the same AWS Region where you plan to execute the action. For more information, seeAdd a cross-Region action in CodePipeline.
As part of creating a pipeline, an S3 artifact bucket provided by the customer will be used by CodePipeline for artifacts. (This is different from the bucket used for an S3 source action.) If the S3 artifact bucket is in a different account from the account for your pipeline, make sure that the S3 artifact bucket is owned by AWS accounts that are safe and will be dependable.
Before you begin this tutorial, you should have already completed the general prerequisites inGetting started with CodePipeline.
Topics
Step 1: Complete prerequisites
To integrate with Jenkins, AWS CodePipeline requires you to install the CodePipeline Plugin for Jenkins on any instance of Jenkins you want to use with CodePipeline. You should also configure a dedicated IAM user or role to use for permissions between your Jenkins project and CodePipeline. The easiest way to integrate Jenkins and CodePipeline is to install Jenkins on an EC2 instance that uses an IAM instance role that you create for Jenkins integration. In order for links in the pipeline for Jenkins actions to successfully connect, you must configure proxy and firewall settings on the server or EC2 instance to allow inbound connections to the port used by your Jenkins project. Make sure you have configured Jenkins to authenticate users and enforce access control before you allow connections on those ports (for example, 443 and 8443 if you have secured Jenkins to only use HTTPS connections, or 80 and 8080 if you allow HTTP connections). For more information, seeSecuring Jenkins.
This tutorial uses a code sample and configures build steps that convert the sample from Haml to HTML. You can download the open-source sample code from the GitHub repository by following the steps inCopy or clone the sample into a GitHub repository. You will need the entire sample in your GitHub repository, not just the .zip file.
This tutorial also assumes that:
You are familiar with installing and administering Jenkins and creating Jenkins projects.
You have installed Rake and the Haml gem for Ruby on the same computer or instance that hosts your Jenkins project.
You have set the required system environment variables so that Rake commands can be run from the terminal or command line (for example, on Windows systems, modifying the PATH variable to include the directory where you installed Rake).
Topics
Copy or clone the sample into a GitHub repository
To clone the sample and push to a GitHub repository
Download the sample code from the GitHub repository, or clone the repositories to your local computer. There are two sample packages:
If you will be deploying your sample to Amazon Linux, RHEL, or Ubuntu Server instances, choosecodepipeline-jenkins-aws-codedeploy_linux.zip.
If you will be deploying your sample to Windows Server instances, chooseCodePipeline-Jenkins-AWSCodeDeploy_Windows.zip.
From the repository, chooseFork to clone the sample repo into a repo in your Github account. For more information, see theGitHub documentation.
Create an IAM role to use for Jenkins integration
As a best practice, consider launching an EC2 instance to host your Jenkins server and using an IAM role to grant the instance the required permissions for interacting with CodePipeline.
Sign in to the AWS Management Console and open the IAM console athttps://console.aws.amazon.com/iam/.
In the IAM console, in the navigation pane, chooseRoles, and then chooseCreate role.
UnderSelect type of trusted entity, chooseAWS service. UnderChoose the service that will use this role, chooseEC2. UnderSelect your use case, chooseEC2.
ChooseNext: Permissions. On theAttach permissions policies page, select the
AWSCodePipelineCustomActionAccessmanaged policy, and then chooseNext: Tags. ChooseNext: Review.On theReview page, inRole name, enter the name of the role to create specifically for Jenkins integration (for example,
JenkinsAccess), and then chooseCreate role.
When you create the EC2 instance where you will install Jenkins, inStep 3: Configure Instance Details, make sure you choose the instance role (for example,JenkinsAccess).
For more information about instance roles and Amazon EC2, seeIAM roles for Amazon EC2,Using IAM Roles to Grant Permissions to Applications Running on Amazon EC2 Instances, andCreating a role to delegate permissions to an AWS service.
Install and configure Jenkins and the CodePipeline Plugin for Jenkins
To install Jenkins and the CodePipeline Plugin for Jenkins
Create an EC2 instance where you will install Jenkins, and inStep 3: Configure Instance Details, make sure you choose the instance role you created (for example,
JenkinsAccess). For more information about creating EC2 instances, seeLaunch an Amazon EC2 instance in theAmazon EC2 User Guide.If you already have Jenkins resources you want to use, you can do so, but you must create a special IAM user, apply the
AWSCodePipelineCustomActionAccessmanaged policy to that user, and then configure and use the access credentials for that user on your Jenkins resource. If you want to use the Jenkins UI to supply the credentials, configure Jenkins to only allow HTTPS. For more information, seeTroubleshooting CodePipeline.Install Jenkins on the EC2 instance. For more information, see the Jenkins documentation forinstalling Jenkins andstarting and accessing Jenkins, as well asdetails of integration with Jenkins inProduct and service integrations with CodePipeline.
Launch Jenkins, and on the home page, chooseManage Jenkins.
On theManage Jenkins page, chooseManage Plugins.
Choose theAvailable tab, and in theFilter search box, enter
AWS CodePipeline. ChooseCodePipeline Plugin for Jenkins from the list and chooseDownload now and install after restart.On theInstalling Plugins/Upgrades page, selectRestart Jenkins when installation is complete and no jobs are running.
ChooseBack to Dashboard.
On the main page, chooseNew Item.
InItem Name, enter a name for the Jenkins project (for example,
MyDemoProject). ChooseFreestyle project, and then chooseOK.Make sure that the name for your project meets the requirements for CodePipeline. For more information, seeQuotas in AWS CodePipeline.
On the configuration page for the project, select theExecute concurrent builds if necessary check box. InSource Code Management, chooseAWS CodePipeline. If you have installed Jenkins on an EC2 instance and configured the AWS CLI with the profile for the IAM user you created for integration between CodePipeline and Jenkins, leave all of the other fields empty.
ChooseAdvanced, and inProvider, enter a name for the provider of the action as it will appear in CodePipeline (for example,
MyJenkinsProviderName). Make sure that this name is unique and easy to remember. You will use it when you add a build action to your pipeline later in this tutorial, and again when you add a test action.This action name must meet the naming requirements for actions in CodePipeline. For more information, seeQuotas in AWS CodePipeline.
InBuild Triggers, clear any check boxes, and then selectPoll SCM. InSchedule, enter five asterisks separated by spaces, as follows:
* * * * *This polls CodePipeline every minute.
InBuild, chooseAdd build step. ChooseExecute shell (Amazon Linux, RHEL, or Ubuntu Server)Execute batch command (Windows Server), and then enter the following:
rakeMake sure that your environment is configured with the variables and settings required to run rake; otherwise, the build will fail.
ChooseAdd post-build action, and then chooseAWS CodePipeline Publisher. ChooseAdd, and inBuild Output Locations, leave the location blank. This configuration is the default. It will create a compressed file at the end of the build process.
ChooseSave to save your Jenkins project.
Step 2: Create a pipeline in CodePipeline
In this part of the tutorial, you create the pipeline using theCreate Pipeline wizard.
To create a CodePipeline automated release process
Sign in to the AWS Management Console and open the CodePipeline console athttp://console.aws.amazon.com/codesuite/codepipeline/home.
If necessary, use the Region selector to change the Region to the one where your pipeline resources are located. For example, if you created resources for the previous tutorial in
us-east-2, make sure that the Region selector is set to US East (Ohio).For more information about the Regions and endpoints available for CodePipeline, seeAWS CodePipeline endpoints and quotas.
On theWelcome page,Getting started page, or thePipelines page, chooseCreate pipeline.
On theStep 1: Choose creation option page, underCreation options, choose theBuild custom pipeline option. ChooseNext.
On theStep 2: Choose pipeline settings page, inPipeline name, enter the name for your pipeline.
CodePipeline provides V1 and V2 type pipelines, which differ in characteristics and price. The V2 type is the only type you can choose in the console. For more information, seepipeline types. For information about pricing for CodePipeline, seePricing.
InService role, chooseNew service role to allow CodePipeline to create a service role in IAM.
Leave the settings underAdvanced settings at their defaults, and chooseNext.
On theStep 3: Add source stage page, inSource provider, chooseGitHub.
UnderConnection, choose an existing connection or create a new one. To create or manage a connection for your GitHub source action, seeGitHub connections.
InStep 4: Add build stage, chooseAdd Jenkins. InProvider name, enter the name of the action you provided in the CodePipeline Plugin for Jenkins (for example
MyJenkinsProviderName). This name must exactly match the name in the CodePipeline Plugin for Jenkins. InServer URL, enter the URL of the EC2 instance where Jenkins is installed. InProject name, enter the name of the project you created in Jenkins, such asMyDemoProject, and then chooseNext.InStep 5: Add test stage, chooseSkip test stage, and then accept the warning message by choosingSkip again.
ChooseNext.
InStep 6: Add deploy stage, reuse the CodeDeploy application and deployment group you created inTutorial: Create a simple pipeline (S3 bucket). InDeploy provider, chooseCodeDeploy. InApplication name, enter
CodePipelineDemoApplication, or choose the refresh button, and then choose the application name from the list. InDeployment group, enterCodePipelineDemoFleet, or choose it from the list, and then chooseNext.You can use your own CodeDeploy resources or create new ones, but you might incur additional costs.
InStep 7: Review, review the information, and then chooseCreate pipeline.
The pipeline automatically starts and runs the sample through the pipeline. You can view progress and success and failure messages as the pipeline builds the Haml sample to HTML and deploys it a webpage to each of the Amazon EC2 instances in the CodeDeploy deployment.
Step 3: Add another stage to your pipeline
Now you will add a test stage and then a test action to that stage that uses the Jenkins test included in the sample to determine whether the webpage has any content. This test is for demonstration purposes only.
If you did not want to add another stage to your pipeline, you could add a test action to the Staging stage of the pipeline, before or after the deployment action.
Add a test stage to your pipeline
Topics
Look up the IP address of an instance
To verify the IP address of an instance where you deployed your code
AfterSucceeded is displayed for the pipeline status, in the status area for the Staging stage, chooseDetails.
In theDeployment Details section, inInstance ID, choose the instance ID of one of the successfully deployed instances.
Copy the IP address of the instance (for example,
192.168.0.4). You will use this IP address in your Jenkins test.
Create a Jenkins project for testing the deployment
To create the Jenkins project
On the instance where you installed Jenkins, open Jenkins and from the main page, chooseNew Item.
InItem Name, enter a name for the Jenkins project (for example,
MyTestProject). ChooseFreestyle project, and then chooseOK.Make sure that the name for your project meets the CodePipeline requirements. For more information, seeQuotas in AWS CodePipeline.
On the configuration page for the project, select theExecute concurrent builds if necessary check box. InSource Code Management, chooseAWS CodePipeline. If you have installed Jenkins on an EC2 instance and configured the AWS CLI with the profile for the IAM user you created for integration between CodePipeline and Jenkins, leave all the other fields empty.
If you are configuring a Jenkins project and it is not installed on an Amazon EC2 instance, or it is installed on an EC2 instance that is running a Windows operating system, complete the fields as required by your proxy host and port settings, and provide the credentials of the IAM user or role you configured for integration between Jenkins and CodePipeline.
ChooseAdvanced, and inCategory, chooseTest.
InProvider, enter the same name you used for the build project (for example,
MyJenkinsProviderName). You will use this name when you add the test action to your pipeline later in this tutorial.This name must meet the CodePipeline naming requirements for actions. For more information, seeQuotas in AWS CodePipeline.
InBuild Triggers, clear any check boxes, and then selectPoll SCM. InSchedule, enter five asterisks separated by spaces, as follows:
* * * * *This polls CodePipeline every minute.
InBuild, chooseAdd build step. If you are deploying to Amazon Linux, RHEL, or Ubuntu Server instances, choose Execute shell. Then enter the following, where the IP address is the address of the EC2 instance you copied earlier:
TEST_IP_ADDRESS=192.168.0.4rake testIf you are deploying to Windows Server instances, chooseExecute batch command, and then enter the following, where the IP address is the address of the EC2 instance you copied earlier:
set TEST_IP_ADDRESS=192.168.0.4rake testChooseAdd post-build action, and then chooseAWS CodePipeline Publisher. Do not chooseAdd.
ChooseSave to save your Jenkins project.
Create a fourth stage
To add a stage to your pipeline that includes the Jenkins test action
Sign in to the AWS Management Console and open the CodePipeline console athttp://console.aws.amazon.com/codesuite/codepipeline/home.
InName, choose the name of the pipeline you created, MySecondPipeline.
On the pipeline details page, chooseEdit.
On theEdit page, choose+ Stage to add a stage immediately after the Build stage.
In the name field for the new stage, enter a name (for example,
Testing), and then choose+ Add action group.InAction name, enter
MyJenkinsTest-Action. InTest provider, choose the provider name you specified in Jenkins (for example,MyJenkinsProviderName). InProject name, enter the name of the project you created in Jenkins (for example,MyTestProject). InInput artifacts, choose the artifact from the Jenkins build whose default name isBuildArtifact, and then chooseDone.Because the Jenkins test action operates on the application built in the Jenkins build step, use the build artifact for the input artifact to the test action.
For more information about input and output artifacts and the structure of pipelines, seeCodePipeline pipeline structure reference.
On theEdit page, chooseSave pipeline changes. In theSave pipeline changes dialog box, chooseSave and continue.
Although the new stage has been added to your pipeline, a status ofNo executions yet is displayed for that stage because no changes have triggered another run of the pipeline. To run the sample through the revised pipeline, on the pipeline details page, chooseRelease change.
The pipeline view shows the stages and actions in your pipeline and the state of the revision running through those four stages. The time it takes for the pipeline to run through all stages will depend on the size of the artifacts, the complexity of your build and test actions, and other factors.
Step 4: Clean up resources
After you complete this tutorial, you should delete the pipeline and the resources it uses so you will not be charged for continued use of those resources. If you do not intend to keep using CodePipeline, delete the pipeline, then the CodeDeploy application and its associated Amazon EC2 instances, and finally, the Amazon S3 bucket used to store artifacts. You should also consider whether to delete other resources, such as the GitHub repository, if you do not intend to keep using them.
To clean up the resources used in this tutorial
Open a terminal session on your local Linux, macOS, or Unix machine, or a command prompt on your local Windows machine, and run thedelete-pipeline command to delete the pipeline you created. For
MySecondPipeline, you would enter the following command:aws codepipeline delete-pipeline --name "MySecondPipeline"This command returns nothing.
To clean up your CodeDeploy resources, follow the instructions inCleaning Up.
To clean up your instance resources, delete the EC2 instance where you installed Jenkins. For more information, seeClean up your instance.
If you do not intend to create more pipelines or use CodePipeline again, delete the Amazon S3 bucket used to store artifacts for your pipeline. To delete the bucket, follow the instructions inDeleting a bucket.
If you do not intend to use the other resources for this pipeline again, consider deleting them by following the guidance for that particular resource. For example, if you want to delete the GitHub repository, follow the instructions inDeleting a repository on the GitHub website.
