Continuous Integration/Delivery in AWS

using Github, Jenkins
with EC2, CodePipeline & CodeDeploy

As one of most important steps in DevOps works, continuous integration/delivery is required to automate for build and deployment process. This part of cycle is usually implemented in cloud platform like Amazon Web Service(AWS). In following article, it'll be showed how to operate all process in AWS using Github and Jenkins tools.

• Launch an EC2 instance make it ready via SSH to install Jenkins
  Firstly we will need an EC2 instance to use as Jenkins virtual server for Build. After logging your AWS account, in AWS Management Console page; from Services click EC2 in Compute header. It will lead you to EC2 dashboard to manage all options related instances.
  
  In EC2 Dashboard, click Launch Instance and choose;
  • Step 1: Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type (free tier eligible)
  • Step 2: t2.micro (free tier eligible / as default)
  • Step 3: IAM role should be assigned to Jenkins instance for interacting with CodePipeline as creating new IAM role. For our instance, other options can be left as default. Choose role you will create by following steps.
    • Create new IAM role - It will lead you Identity and Access Management(IAM) for roles.
    • Create role > AWS service > EC2 > Next: Permissions > AWSCodePipelineFullAccess > Next: Review > Role name = JenkinsCodePipelineRole. Then click Create role.
  • Step 4: Skip this step, no need to edit storage setting for our instance
  • Step 5: Click Add Tag > define your tags as Key = Name and Value = JenkinsServer
  • Step 6: Security Group configuration is significant to allow traffic such as port , IP address etc. to instance to be launched. To connect your instance via SSH, port 22 is required and it's already default setting as rule. To launch JenkinsServer as webserver we will identify new rule in addition to ssh rule by following.
  Assign a security group | Create a new security group > Security group name = ServerSecurity > Click Add Rule > Type : All Traffic > Source : Anywhere > Review and Launch
  • Step 7: Review instance configuration details > Launch

  Once click Launch, Key pair page will pop up. Key pair is required to log into your instance securely from your local, in our case - allow to make SSH connection to JenkinsServer via client.

  • Create a new key pair > Key pair name = sshtoserver > Download Key Pair > Launch Instances > View Instances

  EC2 instance is created to use as Jenkins server, after making ssh connection to instance, Jenkins installation will be done. MobaXterm will be used as ssh client so steps will be executed on that way.

  In MobaXterm Home page, click New session;
  • Basic SSH settings | Remote host = IPv4 Public IP, Specify username=ec2-user, Port=22
    Remote Host is your own IPv4 Public IP number you can provide from Instances section by IPv4 Public IP column. Specify username is ec2-user for Linux AMI that we chose while creating EC2 intance.
  • Advanced SSH settings | Tick Use private key > Select your sshtoserver.pem file from where you downloaded it
  • Network settings | If you are operating these steps from your corporate or other private network, you'll need to use proxy to log in your instance successfully.

  After clicking OK, you will launch your EC2 instance via SSH to install Jenkins with cli prompt.

  In your EC2 instance cli prompt, to install Jenkins run the following commands respectively;

  $ sudo yum update -y 

  $ sudo yum install java-1.8.0 -y

  $ sudo yum remove java-1.7.0-openjdk -y 

  $ sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins.io/redhat/jenkins.repo 

  $ sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key 

  $ sudo yum install jenkins -y 

  $ sudo service jenkins start

  
  Note: EC2 Linux AMI instance is installed with Java 1.7 as default. Jenkins requires Java 1.8 that's why we install java-1.8.0 and remove old version 1.7.0. You can confirm version with 'java -version' command to check it's in required version. After installing new java version; to run it, old version should definitely be removed as we've done

  Now you installed Jenkins on your EC2 instance and ready to login via its web interface.

  • To launch dashboard of Jenkins, in web browser enter your IPv4 Public IP with port 8080 as
    http://IPv4_Public_IP:8080
  • To unlock Jenkins, you need to paste the password value that would be obtained in /var/lib/jenkins/secrets/initialAdminPassword > Continue, use command below to get password
    $ sudo cat /var/lib/jenkins/secrets/initialAdminPassword
  • Customize Jenkins | Install suggested plugins > Create First Admin User > Save and Finish > Instance Configuration (Jenkins URL) > Save and Finish > Start Using Jenkins
  Note: Normally you launch the jenkins dashboard after the latest step however if you meet setup wizard white-blank page; please from command line run commands below respectively to restart jenkins.

  $ sudo service jenkins stop

  $ sudo service jenkins start

• Configure Jenkins to build code before deployment

Jenkins is installed and ready to use, in this phase while plugins over Jenkins are installed and configurations in its interface will be done for build.

In Jenkins Home Page,
• Click Manage Jenkins on the left header, then Manage Plugins
• In Availabe tab, type AWS CodePipeline and search & choose > Download now and install after restart , be sure that select Restart Jenkins when installation is complete and no jobs are running.
• After completing plugin installation; to start your build, click New Item on the left header
• Enter an item name = JenkinsBuild > Freestyle project > OK
• General | Execute concurrent builds if necessary
  Source Code Management | AWS CodePipeline
  > AWS Region is Availability Zone in EC2 Instances page also including in Public DNS; select AWS Region according to your own instance,
  > AWS Access Key & AWS Secret Key are created in Identity and Access Management (IAM), create them by following,
  
  In AWS Management Console,
  • Service > IAM in Security, Identity & Compliance
  • Dashboard > Delete your root access keys > Manage Security Credentials > Continue to Security Credentials
  • Access keys (access key ID and secret access key) > Create New Access Key > Download Key File
  Note: If you don't download your Key file, it won't be possible to check password again. So don't skip downloading.
  
  Then fill values of AWS Access Key & AWS Secret Key obtained by downloading
  > Category : Build,
  > Provider = JenkinsBuildCodePipeline
  Build Triggers | Poll SCM > Schedule = * * * * *
  Build Environment | no need to click any of them, skip
  Build | Add Build Step > Execute Shell
  > In Command section, type following commands respectively as shown below,

  gem install bundle 
  bundle install 
  rake

  Note: Code is Ruby based and ruby is installed as default in our EC2 Linux AMI instance so we don't need to install it again. First two commands above to make ready for environment/workspace; to download Gems in Gemfile 'bundle install' command should be run and to run that related command it should be enable as downloading with 'gem install bundle' command. This is why the first two commands are used and last command 'rake' is used to run script in Rakefile to create index.html we will launch on deployment.
  
  Post-build Actions | Add post-build action > AWS CodePipeline Publisher > Output Locations | Add (click Add), leave Location and Artifact Name as blank
  Save

To run 'rake' command properly, its path should be identified to environment variables in Jenkins.

• Click Manage Jenkins on the left header, then Configure System
• Global properties | tick Environment variables > Add
• List of variables | Name = Path > Value = $PATH:$RAKE_HOME > Add > Name = RAKE_HOME >
  Value = /var/lib/jenkins/bin > Apply > Save

Thus we have done jenkins configuration for build, next step in cycle is deployment which CodeDeploy will be managed.

• Create application & deployment in CodeDeploy to deploy your application

For CodeDeploy, another EC2 instance will be needed to install in and also two different IAM roles should be created and assigned to this new EC2 instance to use CodeDeploy in it and CodeDeploy to allow CodeDeploy to call AWS services on you behalf. To create IAM roles initially;

EC2 instance service role;
AWS Management Console > Services > IAM in Security, Identitiy & Compliance > Roles
• Create Role > EC2 > Next: Permissions
• Search AmazonEC2RoleforAWSCodeDeploy and tick > Next: Review
• Role name = JenkinsEC2CodeDeployRole > Create role

CodeDeploy service role;
• Create Role > CodeDeploy > Select your use case | CodeDeploy > Next: Permissions
• Search AWSCodeDeployRole and tick (it can be assigned automatically) > Next: Review
• Role name = JenkinsCodeDeployRole > Create role

Roles are ready, we can launch a new EC2 instance to install CodeDeploy in it.

• To create & launch EC2 instance please follow Launch an EC2 instance steps also considering next statements;
• Step 3: as IAM role, select JenkinsEC2CodeDeployRole
• Step 5: define your tags as Key = Name and Value = CodeDeployServer
• Step 6: Select an existing security group > select ServerSecurity (same security group) used for also JenkinsServer.
• After Step 7, while launching CodeDeployServer, we can use same key pair created before > Choose an existing key pair > Select a key pair : sshtoserver > tick I acknowledge that statement > Launch Instances
• To launch instance via MobaXterm ssh client, follow same steps indicated before

After clicking OK, you will launch your EC2 instance via SSH.

In your EC2 instance cli prompt, to install CodeDeploy run the following commands respectively;

$ sudo yum update -y

$ wget https://bucket-name.s3.amazonaws.com/latest/install

$ chmod +x ./install

$ sudo ./install auto

$ sudo service codedeploy-agent status


Note: bucket-name is containing AWS CodeDeploy Resource Kit files for your region.For example, for the US East (Ohio) Region, replace bucket-name with aws-codedeploy-us-east-2. Latest command is to check CodeDeploy is up or not. If you see error message like: No AWS CodeDeploy agent running, please run $sudo service codedeploy-agent start command to active it. Also differently from installation document of AWS CodeDeploy in official website, we skip ruby and wget installion since they are installed as default in EC2 instance we have. On the other hand, ec2-user is our default location and user while launching to instance so we didn't need to change directory as shown in offical installion document before wget command.

CodeDeploy installation is completed on EC2 instance and next step is to create application and deployment tasks in CodeDeploy interface.

AWS Management Console > Services > CodeDeploy in Developer Tools > Create application
• Application configuration | Application name = JenkinsCodeDeployApp > Compute platform : EC2/On-premises > Create application
• Create deployment group >
  Deployment group name | Enter a deployment group name = JenkinsCodeDeployment
  Service Role | Choose a service role : JenkinsCodeDeploy (CodeDeploy service role)
  Deployment Type | Choose how to deploy your application : In-place
  Environment configuration | tick Amazon EC2 instances > Value-optional : CodeDeployServer (CodeDeploy-installed EC2)
  Deployment settings | Deployment configuration : CodeDeployDefault.AllAtOnce
  Load balancer | untick Enable load balancing
  Create deployment group

So far we implemented required steps for build and deployment to create a working pipeline. Eventually CodePipeline will help to complete cycle.

• Use CodePipeline to create pipeline for automation with purpose of CI/CD
In our case, we'll use sample code located in Github repository originated by AWS. Before start, logging your Github account, that sample code should be forked (click Fork on upper right) by following link aws-codepipeline-jenkins-aws-codedeploy_linux.zip

Note: Link is directing to my repository I forked from AWS sample code. To be sure it's working on my side so yours, I've added a script (beforeInstall.bash) to solve overwrite issue. Also I want to emphasize that sample code is Linux based as well as EC2 instances operating system.

AWS Management Console > Services > CodePipeline in Developer Tools > Create Pipeline
• Step 1: Choose pipeline settings | Pipeline name = JenkinsCodePipeline > Service Role : New Service role > Role name : AWSCodePipelineServiceRole-(region-name)-JenkinsCodePipeline (it'll be assigned automatically, don't change name manually) > Artifact store : Default location > Next
• Step 2: Source | Source provider : Github > Connect to Github > Repository: entry your own forked repository full name > Branch: Master > Change detection options : Github webhooks (recommended) > Next
• Step 3: Build | Build provider : Add Jenkins > Provider name = JenkinsBuildCodePipeline > Server URL = http://IPv4_Public_IP:8080 > Project name = JenkinsBuild > Next
• Step 4: Deploy | Deployment provider : AWS CodeDeploy > Application name : JenkinsCodeDeployApp > Deployment group : JenkinsCodeDeployment > Next
• Step 5: Review | Create pipeline

After main steps of pipeline, input & output artifacts will be entitled to make source, build and deploy stages successive.

While inside JenkinsCodePipeline; before pipeline starts, click Edit on upper right,
• Edit: Source | Edit stage > > Output artifacts = MyBuild > Save
• Edit: Build | Edit stage > > Input artifacts : MyBuild > Output artifacts = MyDeploy > Save
• Edit: Deploy | Edit stage > > Input artifacts : MyDeploy > Save
• Editing: JenkinsCodePipeline | Save > Save
• Release change > Release

In conclusion, pipeline will start soon according to our last arrangements and stages with all steps we carried out in order. Pipeline visual will show actions with colour highlight for three stages. As process going on smoothly, each stage will turn green colour after completed. With accomplished Deploy stage, application creating index.html webpage is released via IPv4 Public IP of CodeDeployServer instance. As expressed, type IP address in web browser to launch webpage showing greeting text with green background and see result :)
Thanks!