因gitlab-ci的k8s-runner会将job里的每个stage都交由单独的pod来执行，导致默认的本地缓存无法使用。
本文记录了基于k8s pv的gitlab-ci的缓存配置方案，无需使用S3分布式存储。
并记录了gitlab-ci-k8s-runner相关的配置、部署步骤等。

gitlab-ci-k8s-runner部署及缓存配置说明

一 gitlab-ci-cache机制说明

参考：https://blog.csdn.net/xichenguan/article/details/101439395

Gitlab cache 机制可以大大加快 CI/CD Job 的执行速度。基础知识可以参看 Gitlab Cache。
下面直接总结在 Kubernetes 环境中的三种 Cache 的解决方案。

1. Distributed runners caching

gitlab runner job执行前从分布式存储中检查下载解压 cache 文件，job执行后，压缩上传 cache 文件到分布式存储。这是 gitlab 提供的通用的正宗的方法，在非 Kubernetes 环境中也可以使用；

2. 给 Job Executor Pod 挂载同一个 volume

gitlab runner 提供了本地存储 cache 的方式，如果远程存储没有配置，gitlab runner 照样会压缩 cache 文件，然后按照目录规则存储到指定的目录。这种方式的原理是将此目录配置为 Kubernetes Volume，每个 Job 执行时都挂载此 Volume ，这样就相当于所有的 Job 有了一个集中式的存储。可以参考 使用GitLab CI在Kubernetes服务上运行GitLab Runner并执行Pipeline 尝试配置。

3. 不使用缓存，在 Kubernetes 集群中安装各类仓库的私服

Gitlab CI/CI Job 执行时，配置为使用这些私服，速度也很快。

注意：方式1一般指S3存储，建议使用方式2

二 准备缓存pv、pvc

1.配置runner nfs缓存目录

1
2
3
4
5
6
7
export CACHE_DIR=/home/runner-cache
mkdir -p $CACHE_DIR
chmod -R 777 $CACHE_DIR
echo "$CACHE_DIR *(rw,sync,no_root_squash,all_squash,no_subtree_check,anonuid=0,anongid=0)" >> /etc/exports
exportfs -a
showmount -e localhost

2.创建pv、pvc

1
kubectl -n=命名空间 apply -f gitlab-runner-pv-pvc.yaml

3. 创建imagePullSecret

参考：https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account

1
2
3
4
5
kubectl -n=命名空间 create secret docker-registry myregistrykey --docker-server=DUMMY_SERVER \
        --docker-username=DUMMY_USERNAME --docker-password=DUMMY_DOCKER_PASSWORD 
kubectl -n=命名空间 get secret myregistrykey
kubectl -n=命名空间 get secret myregistrykey --output=json

4. 可选：将gitlab-runner、gitlab-runner-helper镜像上传到私有仓库

gitlab-runner镜像

1
2
3
docker tag registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.8.0 私有仓库/gitlab-runner:alpine-v15.8.0
docker push 私有仓库/gitlab-runner:alpine-v15.8.0

gitlab-runner-helper镜像

1
2
3
docker tag registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144 私有仓库/gitlab-runner-helper:x86_64-12335144
docker push 私有仓库/gitlab-runner-helper:x86_64-12335144

三 根据官方helm生成k8s-runner部署文件

参考：https://docs.gitlab.com/runner/install/kubernetes.html
默认values.yaml文件：https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml

1
2
3
4
5
6
helm repo add gitlab https://charts.gitlab.io
helm repo update gitlab
helm install --dry-run --debug --namespace 命名空间 \
gitlab-runner -f values.yaml gitlab/gitlab-runner > gitlab-runner-all.yaml
kubectl -n=命名空间 apply -f gitlab-runner-all.yaml

四 附：docker-runner部署

运行

1
2
3
4
5
docker run -d --name gitlab-runner --restart always \
  -v /home/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

注册

1
2
docker run --rm -it -v /home/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

重启

1
2
3
docker restart gitlab-runner
docker logs -f gitlab-runner

修改配置

在gitlab修改配置runner，支持运行未标记的作业

五 value.yaml和pv-pvc文件参考

gitlab-runner-pv-pvc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gitlab-runner-cache-pv
spec:
  storageClassName: gitlab-runner
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: $NFS_SERVER
    path: "$NFS_PATH"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitlab-runner-cache-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: gitlab-runner
  resources:
    requests:
      storage: 100Gi

gitlab-k8s-runner-helm的value.yaml文件配置

参考：

  1. 默认values.yaml文件：https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml
  2. https://docs.gitlab.com/runner/executors/kubernetes.html#overwriting-container-resources
  3. https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerskubernetes-section
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    image:
      registry: "自行填写"
      image: "自行填写"
      tag: "自行填写"
    imagePullPolicy: IfNotPresent
    imagePullSecrets:
      - name: "自行填写"
    gitlabUrl: "自行填写"
    runnerRegistrationToken: "自行填写"
    # runnerToken: ""
    # unregisterRunners: true
    terminationGracePeriodSeconds: 3600
    # certsSecretName:
    concurrent: 10
    checkInterval: 30
    # logLevel:
    # logFormat:
    # sentryDsn:
    preEntrypointScript: |
      echo "hello"
    sessionServer:
      enabled: false
      # annotations: {}
      # timeout: 1800
      # internalPort: 8093
      # externalPort: 9000
      # publicIP: ""
      # loadBalancerSourceRanges:
      #   - 1.2.3.4/32
    ## For RBAC support:
    rbac:
      create: true
      rules: []
      # - resources: ["configmaps", "pods", "pods/attach", "secrets", "services"]
      #   verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
      # - apiGroups: [""]
      #   resources: ["pods/exec"]
      #   verbs: ["create", "patch", "delete"]
      clusterWideAccess: false
      # serviceAccountName: default
      # serviceAccountAnnotations: {}
      podSecurityPolicy:
        enabled: false
        resourceNames:
          - gitlab-runner
      ## Specify one or more imagePullSecrets used for pulling the runner image
      ##
      ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account
      ##
      imagePullSecrets: []
    metrics:
      enabled: false
      portName: metrics
      port: 9252
      serviceMonitor:
        enabled: false
        ## labels: {}
        # interval: ""
        # scheme: "http"
        # tlsConfig: {}
        # path: "/metrics"
        # metricRelabelings: []
        ## relabelings: []
    service:
      enabled: false
      # labels: {}
      # annotations: {}
      # clusterIP: ""
      # externalIPs: []
      # loadBalancerIP: ""
      # loadBalancerSourceRanges: []
      type: ClusterIP
        # metrics:
      # nodePort: ""
      # additionalPorts: []
    runners:
      # runner configuration, where the multi line strings is evaluated as
      # template so you can specify helm values inside of it.
      #
      # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
      # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
      # 见：1. https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerskubernetes-section
      #  2. https://docs.gitlab.com/runner/executors/kubernetes.html#overwriting-container-resources
      config: |
        [[runners]]
          cache_dir = "/cache"
          [runners.kubernetes]
            namespace = "自行填写"
            image = "自行填写"
            privileged = true
            image_pull_secrets = ["自行填写"]
            image_pull_policy = "if-not-present"
            helper_image = "自行填写"
            helper_cpu_limit = "1000m"
            helper_memory_limit = "2048Mi"
            helper_cpu_requests = "200m"
            helper_memory_requests = "512Mi"
            service_cpu_limit = "1000m"
            service_memory_limit = "2048Mi"
            service_cpu_requests = "200m"
            service_memory_requests = "512Mi"
            cpu_limit = "1000m"
            cpu_limit_overwrite_max_allowed = "5000m"
            memory_limit = "2048Mi"
            memory_limit_overwrite_max_allowed = "10240Mi"
            cpu_requests = "200m"
            cpu_requests_overwrite_max_allowed = "5000m"
            memory_requests = "512Mi"
            memory_requests_overwrite_max_allowed = "10240Mi"
            [[runners.kubernetes.volumes.host_path]]
              name = "docker"
              mount_path = "/var/run/docker.sock"
              read_only = true
              host_path = "/var/run/docker.sock"
            [[runners.kubernetes.volumes.pvc]]
               name = "gitlab-runner-cache-pvc"
               mount_path = "/cache"

    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      privileged: false
      capabilities:
        drop: ["ALL"]
    podSecurityContext:
      runAsUser: 100
      # runAsGroup: 65533
      fsGroup: 65533
      # supplementalGroups: [65533]
      ## Note: values for the ubuntu image:
      # runAsUser: 999
      # fsGroup: 999
    ## Configure resource requests and limits
    ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
    ##
    resources:
       limits:
         memory: 2048Mi
         cpu: 1000m
       requests:
         memory: 512Mi
         cpu: 200m
    affinity: {}
    nodeSelector: {}
    tolerations: []
    # envVars:
    #   - name: RUNNER_EXECUTOR
    #     value: kubernetes
    hostAliases: []
      # Example:
      # - ip: "127.0.0.1"
      #   hostnames:
      #   - "foo.local"
      #   - "bar.local"
      # - ip: "10.1.2.3"
      #   hostnames:
      #   - "foo.remote"
    #   - "bar.remote"
    podAnnotations: {}
      # Example:
    # iam.amazonaws.com/role: <my_role_arn>
    podLabels: {}
      # Example:
      # owner.team: <my_cool_team>
    priorityClassName: ""
    secrets: []
      # Example:
      # - name: my-secret
      # - name: myOtherSecret
      #   items:
      #     - key: key_one
    #       path: path_one
    configMaps: {}
    volumeMounts:
      - name: cache
        mountPath: /cache
    volumes:
      - name: cache
        persistentVolumeClaim:
          claimName: gitlab-runner-cache-pvc