May 7, 2019

Setup your own Jenkins build server and connecting it to GitLab

This post explores how to setup a Jenkins (https://jenkins.io) build server and connect it to a GitLab repository. The GitLab server can be self-hosted or you can make use of https://gitlab.com.

Connecting it to GitLab is optional. If you are using another service or self-host your Git repositories you’ll find instructions on how to remove the GitLab integration at the end of this post.

You can find the code accompanying this post at https://gitlab.com/christian.dev/gitlab-jenkins-starter. It is a simple repository that contains three individual programs writting using Vue.js, Go and Java.

Setup

Jenkins can be setup on a dedicated server or on your development machine. Having it on your computer rather than a central server, allows you to have a fully fledged build server without having to have a separate server.

Install Jenkins

Start by creating the Jenkins container using the following docker-compose.yml file.

version: "3"
services:
  jenkins:
    image: jenkinsci/blueocean
    container_name: jenkins
    restart: always
    privileged: true
    user: root
    ports:
      - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

After running docker-compose up -d wait a moment until you can access Jenkins at http://localhost:8080 (or the port you defined in the docker-compose.yml file).

You need the admin password to access the web UI. The password is randomly generated. You can find it in the log output of the container or by running the command docker exec -it jenkins cat /var/jenkins_home/secrets/initialAdminPassword. (Replace jenkins with whatever you choose for the container_name in the docker-compose.yml or the container id.)

The following Jenkins plugins are required to build our test project. In the Customize Jenkins view make sure that these are selected by choosing the option Select plugins to install.

  • Git plugin

  • Pipeline

  • GitLab
    This plugin is used so the Jenkins server can inform the GitLab server about running builds. For this to work we just have to add the gitLabConnection in the options of the Jenkinsfile. In each build stage we can than add gitlabCommitStatus with the name of the current step. This allows us to see current and past builds in the CI/CD view of GitLab.

    This looks something like this:

    pipeline {
agent {
  docker {
    image 'docker'
  }
}

options {
 gitLabConnection('GitLab')
}

stages {
  stage('Build') {
    parallel {
      stage('Build: webclient-service') {
        steps {
          gitlabCommitStatus(name: 'Build: webclient-service') {
            ...
          }
        }
      }
    }
  }
}

    (See here for the full Jenkinsfile.)

Hit Install and wait for the setup to finish.

Setting up an user is optional. If no user is created the default user name is admin and the password is the same we earlier extracted using the docker exec... command.

The URL configuration on the next screen can be left unchanged. (If you use this guide to setup a central Jenkins server you should but in the actual URL/IP.)

After completing this steps a restart is required. This takes a few moments. (Reload the page if after one or two minutes nothing happened.)

Configure Jenkins

The initial setup is complete and we are almost ready to create the build pipeline. Before doing that we have to tell Jenkins where it can reach GitLab.

In preparation we need to generate a GitLab Access token. Instructions on how to do this can be found here. The scope of the token has to be api.

Navigate to Manage Jenkins/Configure System. There you’ll find a GitLab section. Fill the form as follows:

  • Connection name: The name of the connection as defined in the options/gitLabConnection section of the Jenkinsfile. (In our example “GitLab”.)
  • Gitlab host URL: The base URL of (your) GitLab server. (i.e. https://gitlab.com/)
  • Credentials:
    Choose Add and select Jenkins. A pop-up will open. For Kind select GitLab API token. Paste the earlier created token into the field API token. Give it the ID GITLAB_API_TOKEN and any description. Close the pop-up by clicking Add. The added token can now be selected from the Credentials drop down (found under the given description).

You can check if everything is working by clicking on Test Connection if the output is Success all is good. If not check the error message.

Finish the process by clicking Save on the bottom of the page. You will be redirected to the overview page.

Create build pipeline

A Jenkinsfile is committed in the root folder of the repository. We therefore only have to tell Jenkins where it can find our Git repository.

Choose Open Blue Ocean in the Jenkins overview page. Next click Create a new Pipeline, select Git and paste the URL of the git repository. A new field with the public SSH key for this pipeline will pop up. This key has to be added to the Git server.

Instructions to add a ssh key:

After adding the SSH key to GitLab, finish the pipeline setup by clicking Create Pipeline.

That’s it. The pipeline is automatically triggered and after a few minutes your first build is finished.

Whenever you need a build you can now just open up Jenkins and trigger the pipeline.

The state of your builds is now not only available in Jenkins but also in GitLab. Open your GitLab project an navigate to CI / CD / Pipelines. There you can find for each execution of the pipeline an entry. Clicking it reveals the individual stages. (This is done by adding gitlabCommitStatus... in the Jenkinsfile.)

Trigger builds automatically

The last step is to tell GitLab that the Jenkins server exists. This allows GitLab to trigger a build whenever new code is pushed to the repository.

This step only works if your Jenkins server is available from the Internet or Jenkins and GitLab are on the same (local) network.

For this we first have to activate the Jenkins integration. Open the project in GitLab and navigate to Settings / Integrations in the left menu. Scroll down to find the entry for Jenkins. Opening it reveals the configuration page. Check the boxes for Active and Push and fill in the rest of the fields:

  • Jenkins url: Root URL of your Jenkins server
  • Project name: Choose the same as for the project in GitLab (i.e. gitlab-jenkins-starter)
  • Username: admin (Or any other Jenkins user that is allowed to start builds.)
  • Enter new password: Password for Jenkins user

Save the settings using Test settings and save changes.

With this all is done. Whenever code is pushed to the repository GitLab will trigger Jenkins to start the build.

Remove GitLab integration

If your Git repositories are not hosted at GitLab or you don’t want the integration, you can easily remove it by editing the Jenkinsfile.

  1. Remove the complete options section
  2. Remove all lines starting with gitlabCommitStatus... (Don’t forget to also remove the closing curly bracket })

Bonus (Unittests)

In our example project the individual services are not only build but we also test them using unit tests. The result of theses tests is published to Jenkins. If a test fails the build fails. You can see successful and failed tests in the Tests section of the build pipeline and in each individual build.

Remarks

The Jenkins container has access to the hosts Docker environment. This means that when running the docker command in the Jenkins container it is actually run on the hosts Docker environment. This is important to keep in mind when implementing clean up build steps. Running i.e. docker image prune -a will delete all images on the machine the Jenkins container is running on. I tried using Docker-in-Docker to work around this. But I wasn’t able to get it to work.

If you run Jenkins on your development machine you might wanna have a look on the resource consumption. CPU wise Jenkins only has a significant impact while builds are running. If no builds are running the CPU isn’t used. On the RAM side of things Jenkins will use up some memory even if no builds are running. On my local machine it uses 500MB while idle. If this is a concern for you I recommend to remove the restart option from the Jenkinsfile and only start the Jenkins container when you actually need it.