背景

因为工作需要，需要在一台新集群上部署 jenkins + k8s agent 并结合 gitlab 做一下自动流水线，最后也发部到 k8s 中。

jenkins 部署

helm 仓库

直接使用官方 helm chart 即可，首先添加 helm 仓库

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pv-claim
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-workspace-pv-claim
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

RBAC 定义（用来读取 k8s 的 secret，给动态生成的 runner pod 挂载）：

# role-jenkins-all-secrets-reader.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins-all-secrets-reader
  namespace: jenkins
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"] # 授予 get 和 list 权限
---
# rolebinding-jenkins-all-secrets-reader.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins-default-read-all-secrets
  namespace: jenkins
subjects:
- kind: ServiceAccount
  name: default
  namespace: jenkins
roleRef:
  kind: Role
  name: jenkins-all-secrets-reader # 引用上面的 Role
  apiGroup: rbac.authorization.k8s.io

然后准备好 buildkit 的本地缓存 pvc：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-buildkit-pv-claim
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

最后创建一个 docker 认证密钥

apiVersion: v1
data:
  .dockerconfigjson: xxxxxxx

kind: Secret
metadata:
  name: docker-hub-mereith
  namespace: jenkins
type: kubernetes.io/dockerconfigjson

.dockerconfigjson 是 base64 后的，其具体格式为：

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "xxxxxx"
    }
  }
}

其中 auth 也是一个 base64 编码的，解码后是 username:password 格式的

然后再准备一个 k8s 集群的 config 的 secret，CD 的时候用：

apiVersion: v1
data:
  config: <base64 的 kubeconfig，可以是多上下文的>
kind: Secret
metadata:
  name: kubeconfig
  namespace: jenkins
type: Opaque

这些都准备好后，都 apply 一下让他们生效。

安装

然后开始正式安装 jenkins

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kubectl
    image: lachlanevenson/k8s-kubectl:v1.25.4
    imagePullPolicy: "IfNotPresent"
    command:
    - cat
    tty: true
    volumeMounts:
      - name: kube-config
        mountPath: /root/.kube
        defaultMode: 0600

  - name: buildkit
    image: moby/buildkit:master
    imagePullPolicy: "IfNotPresent"
    command: ["cat"]
    tty: true
    securityContext:
      privileged: true
    volumeMounts:
      - name: buildkit-secret
        mountPath: /.docker
      - name: buildkit-cache
        mountPath: /cache
    env:
      - name: DOCKER_CONFIG
        value: /.docker
  restartPolicy: Never
  volumes:
    - name: buildkit-secret
      secret:
        secretName: docker-hub-mereith
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: kaniko-cache
      persistentVolumeClaim:
        claimName: jenkins-buildkit-pv-claim
    - name: kube-config
      secret:
        secretName: kubeconfig
        items:
          - key: config
            path: config

这个 Raw YAML for the Pod 会合并到默认的模版中，我们添加的文件，会增加两个容器，一个打包用的，一个是发不到 k8s 中。

添加 git 认证密钥

找到 系统管理 -> 凭据管理 -> System -> 全局凭据 -> 添加：

加一个 gitlab api token，去 gitlab 个人账号创建就行。
加一个 username and password，用来认证 git 仓库，当然你用 ssh 也行。

设置 gitlab 插件

找到 系统管理 -> 系统配置 -> GitLab：

添加一个 Connection，密钥就用刚才创建的凭据。

关闭 Crumb 保护

找到 系统管理 -> 脚本命令行:

运行 hudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION = true
关闭同理。
不关也行，按需调整。

流水线配置

新增一个 pipeline 类型的流水线，设置中的 触发器 选 Gitlab，然后按需设置，最后展开 Advanced，生成一个 secret，最上面有一个 webhook 地址。
去 gitlab 项目中的 webhook 设置里添加一个，地址和 secret 都用这个就行。

另外 gitlab 插件默认注入了一些环境变量可以用：

pipeline {
    agent {
        kubernetes {
            inheritFrom 'default' // 确保 'default' Pod 模板存在且包含 'kaniko' (和 'kubectl') 容器
        }
    }
    
    options {
        ansiColor('xterm')
        timeout(time: 20, unit: 'MINUTES') // 全局超时设置
    }

    // --- 参数优化 ---
    // 只定义一个手动触发所需的分支参数
    parameters {
        string(name: 'BRANCH_TO_BUILD', defaultValue: '0414', description: '要构建和部署的 Git 分支')
    }

    environment {
        REGISTRY = "mereith/test"
        IMAGE_NAME = "test"
        DEPLOYMENT_NAME = "test"
        CONTAINER_NAME = "test"
        K8S_NAMESPACE = "app"
        K8S_CONTEXT = "test"
        GIT_REPO_URL = "https://gitlab.example.com/test"
        GIT_CREDENTIALS_ID = "test"
        LARK_WEBHOOK_URL = "xxxx" // 替换为你的飞书 Webhook URL
        BUILD_START_TIME = "${System.currentTimeMillis()}" // 记录构建开始时间
        
        // GitLab 触发时使用的分支名
        BRANCH_NAME = "${env.gitlabBranch ?: params.BRANCH_TO_BUILD}"

        // --- IMAGE_TAG 计算调整 ---
        // 分支名部分，基于输入参数进行清理
        SAFE_BRANCH_NAME = "${env.BRANCH_NAME ?: 'unknown-branch'}".replaceAll('[^a-zA-Z0-9.-]', '-')
        // Commit SHA 部分将在 checkout 后获取，完整的 IMAGE_TAG 将在 Build 阶段构造
        
        // 默认设置初始阶段
        CURRENT_STAGE_NAME = "初始化阶段"
    }

    stages {
        stage('Send Build Triggered Notification') {
            steps {
                script {
                    // 记录当前阶段
                    def stageName = STAGE_NAME
                    env.CURRENT_STAGE_NAME = stageName
                    
                    echo "当前阶段: ${stageName}"
                    echo "发送构建触发通知"
                    
                    // 获取 GitLab 相关信息
                    def gitlabInfo = ""
                    if (env.gitlabActionType) {
                        gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType}
**触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'})
**源分支:** ${env.gitlabSourceBranch ?: 'N/A'}
**目标分支:** ${env.gitlabTargetBranch ?: 'N/A'}"""
                        
                        // 如果是合并请求，添加合并请求信息
                        if (env.gitlabMergeRequestIid) {
                            gitlabInfo += """
**合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}"""
                        }
                    }
                    
                    // 使用上海时区获取当前时间
                    def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai'))
                    
                    // 对特殊字符进行处理，避免JSON格式问题
                    def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"')
                    def jobName = env.JOB_NAME.replaceAll('"', '\\\\"')
                    def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"')
                    def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"')
                    
                    // 发送构建已触发的通知到飞书
                    def triggeredMessage = """{
    "msg_type": "interactive",
    "card": {
        "config": {
            "wide_screen_mode": true
        },
        "header": {
            "title": {
                "tag": "plain_text",
                "content": "🚀 ${jobName} - 构建进行中"
            },
            "template": "blue"
        },
        "elements": [
            {
                "tag": "div",
                "text": {
                    "tag": "lark_md",
                    "content": "**状态:** 构建进行中...\\n**分支:** ${branchName}\\n**构建编号:** #${env.BUILD_NUMBER}\\n**触发时间:** ${shanghaiTime}${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}"
                }
            },
            {
                "tag": "hr"
            },
            {
                "tag": "div",
                "text": {
                    "tag": "lark_md",
                    "content": "**构建详情:** [查看Jenkins任务](${buildUrl}/console)"
                }
            }
        ]
    }
}"""
                    
                    try {
                        // 使用临时文件发送消息，避免命令行参数长度和转义问题
                        writeFile file: 'lark-message.json', text: triggeredMessage
                        sh "curl -s -X POST -H 'Content-Type: application/json' -d @lark-message.json ${env.LARK_WEBHOOK_URL}"
                        sh "rm -f lark-message.json"
                    } catch (Exception e) {
                        env.FAILURE_REASON = "发送构建触发通知失败: ${e.message}"
                        env.FAILED_STAGE = stageName
                        throw e
                    }
                }
            }
        }

        stage('Checkout Source Code') {
            steps {
                script {
                    // 记录当前阶段
                    def stageName = STAGE_NAME
                    env.CURRENT_STAGE_NAME = stageName
                    
                    echo "当前阶段: ${stageName}"
                    
                    try {
                        // 设置较大的 buffer 以解决大型代码库的传输问题
                        def bufferSize = 524288000
                        echo "Setting git config --global http.postBuffer to ${bufferSize}"
                        sh "git config --global http.postBuffer ${bufferSize}"
                        
                        // 使用浅克隆加速检出过程
                        checkout scm: [$class: 'GitSCM',
                            userRemoteConfigs: [[url: env.GIT_REPO_URL, credentialsId: env.GIT_CREDENTIALS_ID]],
                            branches: [[name: env.BRANCH_NAME]],
                            extensions: [
                                [$class: 'CloneOption',
                                    shallow: true,
                                    noTags: true,
                                    depth: 1
                                ]
                            ]
                        ]

                        // 获取实际检出的 Commit SHA
                        env.CHECKED_OUT_COMMIT = sh(script: 'git rev-parse HEAD', returnStdout: true).trim()
                        if (env.CHECKED_OUT_COMMIT.isEmpty()) {
                            env.FAILURE_REASON = "检出代码后无法获取 Commit SHA。"
                            env.FAILED_STAGE = stageName
                            error(env.FAILURE_REASON)
                        }
                        echo "成功检出分支 '${env.BRANCH_NAME}'，Commit SHA: ${env.CHECKED_OUT_COMMIT}"

                        // 获取最新提交者和提交信息
                        env.LATEST_COMMIT_BY = sh(script: 'git log -1 --pretty=format:"%an"', returnStdout: true).trim()
                        env.LATEST_COMMIT_MESSAGE = sh(script: 'git log -1 --pretty=format:"%s"', returnStdout: true).trim()
                        echo "最新提交者: ${env.LATEST_COMMIT_BY}"
                        echo "最新提交信息: ${env.LATEST_COMMIT_MESSAGE}"

                        // 构造最终的 IMAGE_TAG
                        def commitShaShort = env.CHECKED_OUT_COMMIT.take(8)
                        env.FINAL_IMAGE_TAG = "${env.SAFE_BRANCH_NAME}-${commitShaShort}"
                        echo "计算出的镜像 Tag (FINAL_IMAGE_TAG): ${env.FINAL_IMAGE_TAG}"
                    } catch (Exception e) {
                        env.FAILURE_REASON = "代码检出失败: ${e.message}"
                        env.FAILED_STAGE = stageName
                        throw e
                    }
                }
            }
        }
        
        stage('Build and Push Docker Image') {
            steps {
                container('buildkit') {
                    script {
                        // 记录当前阶段
                        def stageName = STAGE_NAME
                        env.CURRENT_STAGE_NAME = stageName
                        env.LAST_ACTIVE_STAGE = stageName
                        
                        echo "当前阶段: ${stageName}"
                        
                        try {
                            // 检查必要的环境变量是否已设置
                            if (env.FINAL_IMAGE_TAG == null || env.FINAL_IMAGE_TAG.isEmpty()) {
                                env.FAILURE_REASON = "FINAL_IMAGE_TAG 未设置，无法构建镜像。"
                                env.FAILED_STAGE = stageName
                                error(env.FAILURE_REASON)
                            }
                            
                            def workspace = pwd()
                            echo "当前目录为 ${workspace}"
                            echo "开始使用 buildkit 构建镜像: ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG}"
                            
                            // 启用缓存使用以加速构建
                            sh """
                            buildctl-daemonless.sh build --local context=${workspace} \\
                                             --local dockerfile=${workspace} \\
                                             --frontend dockerfile.v0 \\
                                             --output type=image,name=${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG},push=true \\
                                             --import-cache type=local,src=/cache \\
                                             --export-cache type=local,dest=/cache
                            """
                            echo "镜像 ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} 构建并推送成功。"
                        } catch (Exception e) {
                            env.FAILURE_REASON = "Docker镜像构建或推送失败: ${e.message}"
                            env.FAILED_STAGE = stageName
                            throw e
                        }
                    }
                }
            }
        }

        stage('Deploy to Kubernetes And Wait Done') {
            steps {
                container('kubectl') {
                    script {
                        // 记录当前阶段
                        def stageName = STAGE_NAME
                        env.CURRENT_STAGE_NAME = stageName
                        env.LAST_ACTIVE_STAGE = stageName
                        
                        echo "当前阶段: ${stageName}"
                        
                        try {
                            // 添加诊断信息
                            try {
                                sh "kubectl version --context ${env.K8S_CONTEXT} --short"
                            } catch (Exception e) {
                                echo "kubectl 版本获取失败，可能是连接或权限问题: ${e.message}"
                            }
            
                            echo "开始部署镜像 ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} 到 Deployment ${env.DEPLOYMENT_NAME}"
                            
                            sh """
                            kubectl set image deployment/${env.DEPLOYMENT_NAME} \\
                                    ${env.CONTAINER_NAME}=${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} \\
                                    -n ${env.K8S_NAMESPACE} \\
                                    --context ${env.K8S_CONTEXT}
                            """
                            
                            // 等待部署完成
                            sh "kubectl --context ${env.K8S_CONTEXT} rollout status deployment/${env.DEPLOYMENT_NAME} -n ${env.K8S_NAMESPACE} --timeout=4m"
                            echo "Deployment ${env.DEPLOYMENT_NAME} 在命名空间 ${env.K8S_NAMESPACE} 中更新成功。"
                        } catch (Exception e) {
                            env.FAILURE_REASON = "Kubernetes 部署超时或失败: ${e.message}"
                            env.FAILED_STAGE = stageName
                            throw e
                        }
                    }
                }
            }
        }
    }
    
    post {
        always {
            echo "Pipeline finished."
        }
        success {
            script {
                echo "Pipeline successfully completed for branch ${env.BRANCH_NAME} at commit ${env.CHECKED_OUT_COMMIT}."
                
                // 计算构建+部署耗时（秒）
                def endTimeMillis = System.currentTimeMillis()
                def durationSeconds = (endTimeMillis - env.BUILD_START_TIME.toLong()) / 1000
                
                // 使用上海时区获取当前时间
                def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai'))
                
                // 获取 GitLab 相关信息
                def gitlabInfo = ""
                if (env.gitlabActionType) {
                    gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType}
**触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'})
**源分支:** ${env.gitlabSourceBranch ?: 'N/A'}"""
                    
                    // 如果是合并请求，添加合并请求信息
                    if (env.gitlabMergeRequestIid) {
                        gitlabInfo += """
**合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}"""
                    }
                }
                
                // 转义特殊字符
                def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"')
                def jobName = env.JOB_NAME.replaceAll('"', '\\\\"')
                def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"')
                def commitMsg = env.LATEST_COMMIT_MESSAGE.replaceAll('"', '\\\\"')
                def commitBy = env.LATEST_COMMIT_BY.replaceAll('"', '\\\\"')
                def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"')
                def imageTag = env.FINAL_IMAGE_TAG.replaceAll('"', '\\\\"')
                
                // 直接发送成功消息
                def successMessage = """{
    "msg_type": "interactive",
    "card": {
        "config": {
            "wide_screen_mode": true
        },
        "header": {
            "title": {
                "tag": "plain_text",
                "content": "✅ ${jobName} - 构建 + 部署成功"
            },
            "template": "green"
        },
        "elements": [
            {
                "tag": "div",
                "text": {
                    "tag": "lark_md",
                    "content": "**分支:** ${branchName}\\n**提交:** ${env.CHECKED_OUT_COMMIT.take(8)}\\n**构建镜像:** ${env.REGISTRY}/${env.IMAGE_NAME}:${imageTag}\\n**提交者:** ${commitBy}\\n**提交信息:** ${commitMsg}\\n**构建+部署耗时:** ${durationSeconds.intValue()} 秒${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}"
                }
            },
            {
                "tag": "note",
                "elements": [
                    {
                        "tag": "plain_text",
                        "content": "构建时间: ${shanghaiTime}"
                    }
                ]
            }
        ]
    }
}"""
                
                try {
                    // 使用临时文件发送消息
                    writeFile file: 'lark-success.json', text: successMessage
                    sh "curl -X POST -H 'Content-Type: application/json' -d @lark-success.json ${env.LARK_WEBHOOK_URL}"
                    sh "rm -f lark-success.json"
                } catch (Exception e) {
                    echo "发送成功通知失败: ${e.message}"
                }
            }
        }
        failure {
            script {
                echo "Pipeline failed for branch ${env.BRANCH_NAME}."
                
                // 计算构建+部署耗时（秒）
                def endTimeMillis = System.currentTimeMillis()
                def durationSeconds = (endTimeMillis - env.BUILD_START_TIME.toLong()) / 1000
                
                // 使用上海时区获取当前时间
                def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai'))
                
                // 获取失败的阶段信息，使用环境变量而不是通过不安全的API
                def failedStage = env.FAILED_STAGE ?: env.LAST_ACTIVE_STAGE ?: env.CURRENT_STAGE_NAME ?: "未知阶段"
                def failureReason = env.FAILURE_REASON ?: "未知原因"
                
                // 获取 GitLab 相关信息
                def gitlabInfo = ""
                if (env.gitlabActionType) {
                    gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType}
**触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'})
**源分支:** ${env.gitlabSourceBranch ?: 'N/A'}"""
                    
                    // 如果是合并请求，添加合并请求信息
                    if (env.gitlabMergeRequestIid) {
                        gitlabInfo += """
**合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}"""
                    }
                }
                
                // 转义特殊字符
                def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"')
                def jobName = env.JOB_NAME.replaceAll('"', '\\\\"')
                def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"')
                def commitMsg = (env.LATEST_COMMIT_MESSAGE ?: 'N/A').replaceAll('"', '\\\\"')
                def commitBy = (env.LATEST_COMMIT_BY ?: 'N/A').replaceAll('"', '\\\\"')
                def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"')
                
                // 在消息中记录调试信息
                echo "调试信息 - FAILED_STAGE: ${env.FAILED_STAGE}, CURRENT_STAGE_NAME: ${env.CURRENT_STAGE_NAME}, LAST_ACTIVE_STAGE: ${env.LAST_ACTIVE_STAGE}"
                
                // 直接发送失败消息
                def failureMessage = """{
    "msg_type": "interactive",
    "card": {
        "config": {
            "wide_screen_mode": true
        },
        "header": {
            "title": {
                "tag": "plain_text",
                "content": "❌ ${jobName} - 构建失败"
            },
            "template": "red"
        },
        "elements": [
            {
                "tag": "div",
                "text": {
                    "tag": "lark_md",
                    "content": "**分支:** ${branchName}\\n**提交:** ${env.CHECKED_OUT_COMMIT ? env.CHECKED_OUT_COMMIT.take(8) : 'N/A'}\\n**提交者:** ${commitBy}\\n**提交信息:** ${commitMsg}\\n**失败阶段:** ${failedStage}\\n**失败原因:** ${failureReason}\\n**构建+部署耗时:** ${durationSeconds.intValue()} 秒${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}"
                }
            },
            {
                "tag": "div",
                "text": {
                    "tag": "lark_md",
                    "content": "**查看详情:** [Jenkins构建日志](${buildUrl}/console)"
                }
            },
            {
                "tag": "note",
                "elements": [
                    {
                        "tag": "plain_text",
                        "content": "构建时间: ${shanghaiTime}"
                    }
                ]
            }
        ]
    }
}"""
                
                try {
                    // 使用临时文件发送消息
                    writeFile file: 'lark-failure.json', text: failureMessage
                    sh "curl -X POST -H 'Content-Type: application/json' -d @lark-failure.json ${env.LARK_WEBHOOK_URL}"
                    sh "rm -f lark-failure.json"
                } catch (Exception e) {
                    echo "发送失败通知失败: ${e.message}"
                }
            }
        }
    }
}