Experience with GitLab CI, Docker and Kubernetes Pipelines — Part 2

Welcome back! Assuming that you have already read my first blog on GitLab installation and configuration, let’s move to the next phase on configuring and creating a GitLab Runner and a .gitlab-ci.yaml file respectively.

GitLab Runner

What is a GitLab Runner and Executor?

A GitLab Runner runs the pipeline jobs and then sends the result back to GitLab, in other words, a slave node that runs the CI processes on it and sends feedback to the master node.

Executors are the implementers on GitLab Runners that are used to run the builds in different scenarios.

Initially, when I created a GitLab Runner using Kubernetes executor as a pod, there was a security risk to perform Docker builds in the Kubernetes cluster. To perform Docker builds in GitLab Runner pod, the /var/run/docker.sock has to be mounted as a host volume to share the docker daemon and give privileged access (root access) to the pod. If the GitLab runner pod gets compromised, not only the underlying Kubernetes worker node host but also the cluster gets compromised.

Therefore, I decided to create two GitLab Runners with Docker executors in a single VM/instance. That being said, I created an ec2-instance with minimal permissions to access other AWS services using IAM policies and security groups. By using the gitlab-runner binary, I registered and installed two GitLab Runners with Docker executors.

The Docker executor in the ec2-instance performs the Docker builds by mounting the /var/run/docker.sock of the host. Although this method avoids using Docker in privileged mode eliminating the risk of compromising the hosts, there are some implications that you should be aware of when using this approach.

Runner Installation

To install GitLab Runner, I created a t2.medium sized ec2-instance with an Ubuntu Amazon Machine Image (AMI).

Once the instance is in the ready state, the following commands are to install Docker,

# Remove old Docker components
sudo apt-get remove docker docker-engine docker.io containerd runc
# Update system
sudo apt-get update
# Install some common packages
sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
# Add Docker key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# Add Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# Update system
sudo apt-get update
# Install Docker
sudo apt-get install docker-ce docker-ce-cli containerd.io
# Check Docker status
systemctl status docker

Once the Docker is up and running, install the gitlab-runner and start the service with the following commands,

# Download Gitlab package
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
# Install GitLab Runner
sudo apt-get install gitlab-runner
# Check the GitLab runner installation
gitlab-runner version
# Check the status of GitLab runner
systemctl status gitlab-runner

Assigning Runner to a GitLab project

After the Gitlab Runner is installed, I registered the first GitLab Runner to my sample application project in GitLab to perform the Maven build and test. Now let’s run the following command to register and assign a GitLab Runner with a Docker executor.

# Register the First Gitlab Runner with the options
sudo gitlab-runner register --url "https://gitlab.example-labs.com" --registration-token "xxxxxxxxxxxxxxxx" --docker-image "docker:stable" --tag-list default, docker --executor docker --tls-cert-file gitlab.example-labs.crt

Here are the details for the options in the above command,

GitLab URL and Token at projects>{Project-Name}>settings>CI/CD>Runners page

url: The GitLab

url: https://gitlab.example-labs.com

registration-token: Path to registration token in projects>demo-java>settings>CI/CD>Runners

docker-Image: I used “docker:stable” docker image for both the executors.

--docker-image: "docker:stable"

tag-list: Tagging the runner executor is very helpful when building a stage with an executor.

--tag-list: default,docker

executor: For the demo-java application, I used docker executor to run every stage in the pipeline, as docker containers.

--executor: docker

tls-cert-file: Since executor connects to the GitLab instance using https, the tls-cert-file of the GitLab instance should be specified while running the register command.

--tls-cert-file: gitlab.example-labs.crt

Note: To copy the GitLab instance certificate, login to GitLab instance and change directory to /etc/gitlab/ssl and copy the certificate file with the <hostname>.crt and paste it in the Gitlab Runner Machine with the same name as <hostname>.crt, (In my case, it was gitlab.example-labs.com.crt) and specify it with the relative/absolute path in the register command.

Output:
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Make sure the config.toml file in /etc/gitlab-runner/config.toml is similiar to the one below,

concurrent = 1
check_interval = 0
[session_server]
  session_timeout = 1800
[[runners]]
  name = "default"
  url = "https://gitlab.example-labs.com/"
  token = "xxxxxxxxxxxxxxx"
  tls-cert-file = "gitlab.example-labs.com.crt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.custom]
    run_exec = ""

Check the GitLab Runner status in GitLab instance.

GitLab-Runner in GitLab

Now that I registered the first GitLab Runner for Maven builds and test, I used the same command as above with minor changes in the token and tags to create the second GitLab Runner in the same ec2-instance with /var/run/docker.sock as volume to perform Docker builds and push.

# Register the Second Gitlab Runner with the options
sudo gitlab-runner register --url "https://gitlab.example-labs.com" --registration-token "xxxxxxxxxxxxxxxx" --docker-image "docker:stable" --tag-list docker-build --executor docker --tls-cert-file gitlab.example-labs.crt --docker-volumes /var/run/docker.sock:/var/run/docker.sock

Check the GitLab instance for the other executor in the Runners section.

Two Runners are active in GitLab instance

Below is the final config.toml file with two GitLab Runners in /etc/gitlab-runner/config.toml

concurrent = 1
check_interval = 0
[session_server]
  session_timeout = 1800
[[runners]]
  name = "default"
  url = "https://gitlab.example-labs.com/"
  token = "xxxxxxxxxxxxxxxx"
  tls-cert-file = "gitlab.example-labs.com.crt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.custom]
    run_exec = ""
[[runners]]
  name = "My Docker Runner"
  url = "https://gitlab.example-labs.com/"
  token = "xxxxxxxxxxxxxxxx"
  tls-cert-file = "gitlab.example-labs.com.crt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.custom]
    run_exec = ""

Congratulations!!! We have finally created two GitLab Runners. Now’s let’s figure out the .gitlab-ci.yaml file for the demo-java application.

.gitlab-ci.yaml

.gitlab-ci.yaml are a yaml structured files which are used to configure CI/CD pipelines in GitLab project.

demo-java project with .gitlab-ci.yaml file

Let’s look at my .gitlab-ci.yaml (comments are inplace for better understanding)

# Stages for my CI/CD Pipeline
stages:
  - build
  - test
  - image-build
  - deploy-dev

# Global Variable 
variables:
    IMAGE_NAME: gitlabci-app


# Build stage with maven docker image and uploading artifacts to 
# Gitlab CI 
build:
  image: 
    name: maven:3.6.1-jdk-14
  stage: build
  # using the first runner with docker tag in build stage
  tags: 
    - docker       
  script:
    - mvn package
  artifacts:
    paths:
      - target/demo.war
    expire_in: 1 week


# Test stage with maven docker image
test:
  image: maven:3.6.1-jdk-14
  stage: test
  script:
    - mvn test
  # using first runner with docker tag in Test stage
  tags: 
    - docker


# image-build stage for Docker build and Dokcer push the 
# maven artifacts
image-build:
  image:
    name: docker:stable
  stage: image-build
  # using second runner with docker build tag to perform docker
  # commands
  tags:
    - docker build  
  script:
    - docker login -u $REPOSITORY_NAME -p $PASSWORD
    - docker build -t $REPOSITORY_NAME/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA .
    - docker push $REPOSITORY_NAME/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA
  # downloading artifact from build stage
  dependencies:
    - build

# Deply the Docker image to kubernetes(EKS) cluster
deploy-to-dev:
  image:
    # customized docker image to setup kubeconfig and run kubectl                    
    name: suryalolla/kubectl-docker:0.0.1
  stage: deploy-dev
  # using second runner with docker build tag to perform docker 
  # commands
  tags:
    - docker build
  variables:
    DEPLOYMENT_NAME: web-application
    SERVICE_NAME: web-application-service
    KUBE_NAMESPACE: dev-env
    APP_NAME: demo-app
    APP_REPLICAS: 2
    AWS_REGION: us-east-1
    CLUSTER_NAME: EKS-example
  script:
    - aws eks --region $AWS_REGION update-kubeconfig --name $CLUSTER_NAME
    - chmod +x templates/*
    - ./templates/deployment-template.sh true $DEPLOYMENT_NAME $KUBE_NAMESPACE $APP_NAME $APP_REPLICAS $REPOSITORY_NAME $IMAGE_NAME 
    - ./templates/service-template.sh true $SERVICE_NAME $KUBE_NAMESPACE $APP_NAME
    - kubectl apply -f deployment.yaml
    - kubectl apply -f service.yaml

In the above .gitlab-ci.yaml file, I created four stages: build, test, image-build and deploy-to-dev.

The first two stages are pretty straightforward, as it uses maven docker image to build and test the Java application. Since we used the docker tag in both the stages, the application build and test are executed only on the first runner.

The third i.e., image-build stage, is all about building the application docker image and pushing it into a private docker repository in docker:stable container. Since the /var/run/docker.sock is mounted to the second runner, I specified it (second runner) as docker build tag in .gitlab-ci.yaml file.

In the CI/CD pipeline configuration, I defined variables section in the .gitlab-ci.yaml to store the shared values and also added some sensitive variables like passwords and certs in GitLab UI.

Variable in GitLab UI

# Vars in .gitlab-ci.yaml
variables:
  DEPLOYMENT_NAME: web-application
  SERVICE_NAME: web-application-service
  KUBE_NAMESPACE: dev-env
  APP_NAME: demo-app
  APP_REPLICAS: 2
  AWS_REGION: us-east-1
  CLUSTER_NAME: EKSexample

deploy-to-dev stage performs three tasks in suryalolla/kubectl-docker:0.0.1 container to deploy the application image to an EKS cluster,

1. First, it generates a kubeconfig for an EKS cluster using aws command.
2. It runs the shell scripts in the templates directory to create the Kubernetes manifest files.
3. To deploy the application image on an EKS cluster, it uses the above generated kubeconfig and performs the kubectl apply commands on the kubernetes manifest files.

Note: In the deploy-to-dev stage, I used a customized docker image to setup aws and kubectl CLI. Feel free to modify the Dockerfile and create your own image with AWS credentials in it.

Once you commit and push the .gitlab-ci.yaml file into the demo-java project, it should trigger a pipeline and run all the stages accordingly.

Since the Auto-Devops option is enabled by default in the GitLab, any new commits to the repository will trigger the GitLab CI pipeline.

GitLab CI pipeline for demo-java application

Yay!! Finally got the GitLab CI pipeline working with Maven, Docker and Kubernetes.

Conclusion

Overall, I had a great experience with GitLab and I feel it is a great tool to explore. It is easy to write pipelines using YAML files compared to Jenkins Groovy scripts and requires no additional Version Control configuration.

Here’s the GitHub repo for your reference,

venkatalolla/demo-java

Hope you liked it 🙃

Happy Automation 🚀

Until next time!!!