9 minute read

All pages

Name Summary
Monitoring with Prometheus, Loki, Grafana and Kubernetes. Part 1. Kubernetes cluster Here is about basic configuration of Kubernetes monitoring cluster
Monitoring with Prometheus, Loki, Grafana and Kubernetes. Part 2. SNMP Here is about SNMP O_O
Monitoring with Prometheus, Loki, Grafana and Kubernetes. Part 3. GitLab Agent How to connect a Kubernetes cluster to GitLab
Monitoring with Prometheus, Loki, Grafana and Kubernetes. Part 4. Prometheus exporters Exporting Prometheus metrics
Monitoring with Prometheus, Loki, Grafana and Kubernetes. Part 5. kube-prometheus-stack Migration to kube-prometheus-stack

In this part, I wanted to write about the SNMP Exporter, but I decided, that it’s time to think about automating changes in the cluster. Therefore, here I will connect the Kubernetes cluster to Gitlab using the Gitlab Agent. The result will be the ability to manage the cluster using the Gitlab pipeline. All this in the “How to” format .

Now I have a Kubernetes cluster, deployed on a Desktop machine (see Kubernetes manifests from Part 1. This is a development environment. This is where I constantly make changes and apply them manually using kubectl apply. Automating changes, using Gitlab + Gitlab Agent, in the development cluster is redundant, so I will create a virtual machine on Ubuntu Server 22.04 and deploy a Production Kubernetes cluster on it, which I will connect to Gitlab. I will use a non-public network and self-hosted services.

So, I build the local environment on the Production machine (new machine on Ubuntu 22.04):

Besides the new Production cluster, I need Gitlab and DNS. I already have a Gitlab 14.8 Omnibus installation, deployed on a separate machine, which is available at http://gitlab.my.domain. In Gitlab I created a k8s repository and added Kubernetes manifests from Part 1 to it.

k8s - directory with manifests (grafana, loki, prometheus contain corresponding manifests):

ingress - directory with Ingress file:

Kubernetes and Gitlab integration

https

I start with this link. Here, the Gitlab Agent should interact with the Gitlab Kubernetes Agent Server (KAS) only via https, because credentials are not transmitted on http, and kubectl command, running in the Gitlab pipeline, will show this error: error: You must be logged in to the server (the server has asked for the client to provide credentials).

Certificates

So, a certificate is a public key signed with the private key of a certification authority. To sign, you can:

  • use a public certification authority, for example, LetsEncrypt
  • create your own certificate authority
  • use the private key that was used to generate the public key (without CA). It’s self-signed certificate

The public part of the private key, used to sign the certificate, must be added to all web browsers that will trust the certificate.

So, I connect to the machine with Gitlab Omnibus, here I will create a certificate for Gitlab (this option is suitable for simple installations or educational purposes). This certificate will be used by Gitlab and Gitlab Kubernetes Agent Server (KAS). Gitlab will work on https.

mkdir ~/certs && cd ~/certs

# create a private key with a length of 2048 bits
openssl genrsa -out gitlab.my.domain.key 2048

# create a certificate signing request (csr) to get a certificate
# write Common Name (CN) field only - gitlab.my.domain as an example
openssl req -key gitlab.my.domain.key -new -out gitlab.my.domain.csr

Now I need to sign the received .csr file with the private key of the certification authority. Since I don’t have a public Gitlab, I won’t be using public CAs (but this option can be used for non-public services). I can either sign the .csr file with the private key I created earlier (gitlab.my.domain.key) or create my own CA. For this task, the first option is fine, but I like the second one better.

I create my own certificate authority(CA). This is the directory with the self-signed CA certificate:

mkdir ~/ca && cd ~/ca

# create a private key, certificate signing request 
# and a public key (self-signed CA certificate) with one command 
openssl req -newkey rsa:2048 -nodes -keyout ca.key -x509 -days 3654 -out ca.crt

I sign the certificate signing request (.csr file) received earlier. From a certain version, Kubernetes considers the Common Name (CN) field as legacy and asks to use SAN (subjectAltName), so I add this field (SAN) to the certificate:

openssl x509 -req -extfile <(printf "subjectAltName=DNS:gitlab.my.domain") \
-in ~/certs/gitlab.my.domain.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out gitlab.my.domain.crt -days 365

3654 and 365 - lifetime of certificates. Here as an example. It is logical that CA has a larger number. And now I have a certificate for Gitlab + Gitlab KAS - gitlab.my.domain.crt.

Gitlab + https

Now, I have the certificate (public key - gitlab.my.domain.crt file) and private key (gitlab.my.domain.key), I will configure Gitlab to work with https.

The Gitlab configuration is stored in the /etc/gitlab/gitlab.rb file. I do according to the manual:

# open Gitlab configuration file in vim editor
sudo vi /etc/gitlab/gitlab.rb

# configure this options
external_url "https://gitlab.my.domain"
letsencrypt['enable'] = false

# enable redirect from http to https:
nginx['redirect_http_to_https'] = true

# see this issue - https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7356
gitlab_kas['env'] = {
    'SSL_CERT_DIR' => '/opt/gitlab/embedded/ssl/certs'
}

# copy private key and certificate to /etc/gitlab/ssl
sudo mkdir -p /etc/gitlab/ssl
sudo chmod 755 /etc/gitlab/ssl
sudo cp gitlab.my.domain.key gitlab.my.domain.crt /etc/gitlab/ssl/

# copy certificate to /etc/gitlab/trusted-certs
# see this issue - https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7356
sudo cp gitlab.my.domain.crt /etc/gitlab/trusted-certs/

Allow https in iptables, if needed, and reconfigure gitlab:

sudo gitlab-ctl reconfigure

Gitlab will now be available at https://gitlab.my.domain. And here is usefull link for troubleshooting SSL

Gitlab Runner

To run the Gitlab pipeline, I need to install Gitlab Runner. There can be many Gitlab Runners, it is recommended to install them on separate machines, not on the Gitlab machine. For this tutorial, I will make an exception to the rule and register 1 runner on the Gitlab machine. Documentation has many installation options, I will choose Docker

First, I need to generate a token for the new runner. Log in to Gitlab UI as root > Configure GitLab > Features > Shared Runners > Register an Instance Runner > Copy registration token.

Next, I add the public part of the previously generated ca.crt key to the /srv/gitlab-runner/config/certs/ directory (Gitlab Runner will work over https and must trust my ca certificate). The directory can be changed. This directory will be mounted into the Gitlab Runner container:

# copy ca.crt to certs directory 
sudo cp ~/ca/ca.crt /srv/gitlab-runner/config/certs/

# install ant register a new Gitlab Runner
docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register
# Gitlab server - https://gitlab.my.domain
# paste the preveously generated token from web UI Gitlab
# runner name - docker-runner0
# tag to run in the Gitlab pipeline - docker

Now the new Runner will appear in the Gitlab web UI, the status should be online.

Kubernetes Agent Server and Kubernetes Agent

Kubernetes Agent Server (KAS)

At first, enable Kubernetes Agent Server on Gitlab server (Docs for Omnibus installation):

sudo vi /etc/gitlab/gitlab.rb

# configure /etc/gitlab/gitlab.rb 
gitlab_kas['enable'] = true

Reconfigure Gitlab:

sudo gitlab-ctl reconfigure

Gitlab Agent

And now it’s time to install the Gitlab Agent to my Kubernetes Production cluster. Here is link from docs step by step:

  • Create an agent configuration file .gitlab/agents/k8s/config.yaml in default branch k8s Gitlab repository with Gitlab UI

  • Authorize the agent to access your projects (if the project is the only one - this is optional, see docs) with .gitlab/agents/k8s/config.yaml file:

      # add to .gitlab/agents/k8s/config.yaml
  ci_access:
    projects:
      - id: my_gitlab_username/project

  • Register the agent with GitLab UI. Select k8s Gitlab project > Infrastructure > Kubernetes clusters > Actions > Connect with Agent > Select an Agent (k8s agent) > Register Agent and copy the command under “Recommended installation method”. For my Gitlab 14.8 it looks like:

      docker run --pull=always --rm \
  registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate \
  --agent-token=my_token \
  --kas-address=wss://gitlab.my.domain//-/kubernetes-agent \
  --agent-version stable \
  --namespace gitlab-kubernetes-agent

But, before installing gitlab-agent in the Production cluster with this command, I need to add ca.crt to the list of trusted certificates. This issue describes how this can be done. I’ll just repeat on a machine with a Production Kubernetes cluster:

# create generic secret with encoded ca.crt
kubectl create secret generic 'ca' --namespace gitlab-kubernetes-agent --from-file="~/crt.ca"

# save kas configuration to kas.yaml
docker run --pull=always --rm \
registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate \
--agent-token=my_token \
--kas-address=wss://gitlab.my.domain//-/kubernetes-agent \
--agent-version stable \
--namespace gitlab-kubernetes-agent > kas.yaml

# open kas.yaml with vi
vi kas.yaml

# search kind: Deployment section 
# add ca.crt to spec:template:spec:containers:args
spec:
  containers:
  - args:
    - --token-file=/config/token
    - --kas-address
    - wss://gitlab.my.domain/-/kubernetes-agent/
    - --ca-cert-file=/certs/ca.crt

# add new volume with ca.crt secret object to spec:template:spec:volumes
volumes:
- name: custom-certs
  secret:
    secretName: ca

# mount new custom-certs volume to /certs directory 
# in spec:template:spec:contaiers:volumeMounts section 
volumeMounts:
- name: custom-certs
  readOnly: true
  mountPath: /certs

# save and close kas.yaml with vi
:wq

# apply file with kubectl
kubectl apply -f kas.yaml

Now the agent status in Gitlab UI (k8s project > Infrastructure > Kubernetes clusters) will change to Connected.

Gitlab pipeline

Now my k8s repository contains Kubernetes manifests and authorization file for Gitlab Agent. I will add new .gitlab-ci.yml file in k8s repo with Gitlab UI.

.gitlab-ci.yml here is a simple 3 stage pipeline file (validate, deploy, report) just for example and testing the work Gitlab Agent with Kubernetes cluster:

stages:
    - validate
    - deploy 
    - report 

default:
  tags:
    - docker

variables:
  KUBE_CONTEXT: my_gitlab_user/k8s:k8s

before_script:
  - kubectl config get-contexts
  - kubectl config get-clusters
  - kubectl config set-cluster gitlab --certificate-authority="$KAS_CA"
  - if [ -n "$KUBE_CONTEXT" ]; then kubectl config use-context "$KUBE_CONTEXT"; fi

lint-kubeconform:
  stage: validate
  image:
    name: ghcr.io/yannh/kubeconform:latest-alpine
    entrypoint: [""]
  before_script:
    - ''
  script:
  - ~/../kubeconform ./k8s/

deploying:
  stage: deploy
  image:
    name: bitnami/kubectl:latest
    entrypoint: [""]
  script:
  - kubectl apply -R -f ./k8s

viewing-resources:
  stage: report
  image:
    name: bitnami/kubectl:latest
    entrypoint: [""]
  script:
  - kubectl get deployment,pod,services,cm,pv,secrets -A
  - kubectl events -A

About this pipeline:

  • pipeline contains 3 stages: validate, deploy, report. Validate stage uses kubeconform validation tool. Here I specified a k8s directory, that does not contain Kubernetes manifest (the k8s directory contains subdirectories with Kubernetes manifests), so validation will always succeed. Validation is not the purpose of this “how to” and is just an example here
  • tags: docker runs pipeline in previously configured Gitlab runner
  • variables: KUBE_CONTEXT: my_gitlab_user/k8s:k8s - the result of the kubectl config get-contexts command running in the pipeline (first k8s is the name of the repository, second k8s is the name of the directory with the config.yaml agent file)
  • the most interesting is in the before_script section. Gitlab Agent can communicate with Gitlab KAS used ca.crt certificate, but kubectl, executed in pipeline, will fail to communicate with KAS, because k8s uses an its own self-generated certificate by default (check KUBECONFIG configuration with kubect config view command) and k8s client doesn’t trust the certificate authority (CA) that signed KAS certificate. For example, kubectl get pods command, running in a pipeline, will show the following error: 
      $ kubectl get pods
  Unable to connect to the server: x509: certificate signed by unknown authority

    See docs. I like the solution from this issue. KUBECONFIG in the pipeline must trust KAS ca.crt. Following this instruction, I created CI/CD variable with KAS_CA name and added the content of ca.crt there. The kubectl config set-cluster gitlab --certificate-authority="$KAS_CA" command will add ca.crt to the list oftrusted certificates in the gitlab cluster before deploy and report stages.

    Other related issues:

Now, if I update the Kubernetes manifests in the Development environment and push changes to Gitlab, the Gitlab pipeline will update the Production Kubernetes cluster

Troubleshooting

How to check KAS logs (for Gitlab Omnibus):

gitlab-ctl tail gitlab-kas
# or
sudo tail -f /var/log/gitlab/gitlab-kas/current

How to check Gitlab Kubernetes Agent logs:

kubectl logs -f -l=app=gitlab-agent -n gitlab-kubernetes-agent

And docs:

How to “git clone”

Since my Gitlab installation uses a self-signed certificate, “git clone” for https k8s repository might look like this:

GIT_SSL_NO_VERIFY=true git clone https://gitlab.my.domain/my_gitlab_user/k8s.git

cd k8s
git config --local http.sslVerify "false"

You may also enjoy