自动化构建 Java 应用

2023-10-18

字数统计: 4k字 | 阅读时长≈ 20分

资料来源: Kubernetes全栈架构师：基于世界500强的k8s实战课程

自动化构建 Java 应用

创建 Java 测试用例

示例项目可以从 https://gitee.com/dukuan/spring-boot-project.git 找到该项目(也可以使用公司的 Java 项目也是一样的)。接下来将该项目导入到自己的 Gitlab 中。

首先在 GitLab 中创建 kubernetes 组，然后在 kubernetes 组中点击 New Projects, 选择 Import Project，点击 Repo by URL。在 Git repository URL 处输入示例地址，然后点击 Create Project 即可。创建好的代码仓库如下所示:

定义 Jenkinsfile

接下来再 GitLab 的源代码中添加 Jenkinsfile。

首先点击代码首页的“+”号，然后点击 New file，输入文件名为 Jenkinsfile:

在窗口中添加以下内容:

pipeline {
  agent {
    kubernetes {
      cloud 'kubernetes-test'
      slaveConnectTimeout 1200
      workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
      yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
  - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
    image: \'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine\'
    name: jnlp
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
  - command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"
    imagePullPolicy: "IfNotPresent"
    name: "build"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
    - mountPath: "/root/.m2/"
      name: "cachedir"
      readOnly: false
  - command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
    imagePullPolicy: "IfNotPresent"
    name: "kubectl"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
  - command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
    imagePullPolicy: "IfNotPresent"
    name: "docker"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "dockersock"
      readOnly: false
  restartPolicy: "Never"
  nodeSelector:
    build: "true"
  securityContext: {}
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "dockersock"
  - hostPath:
      path: "/usr/share/zoneinfo/Asia/Shanghai"
    name: "localtime"
  - name: "cachedir" 
    hostPath:
      path: "/opt/m2"
'''
    }
  }
  stages {
    stage('Pulling Code') {
      parallel {
        stage('Pulling Code by Jenkins') {
          when {
            expression {
              env.gitlabBranch == null
            }
          }
          steps {
            git(
              url: 'git@gitlab.59izt.com:kubernetes/spring-boot-project.git', 
              changelog: true, 
              poll: true, 
              branch: "${BRANCH}", 
              credentialsId: 'gitlab-key'
            )
            script {
              COMMIT_ID = sh(
                returnStdout: true, 
                script: "git log -n 1 --pretty=format:'%h'"
              ).trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }
          }
        }

        stage('Pulling Code by trigger') {
          when {
            expression {
              env.gitlabBranch != null
            }
          }
          steps {
            git(
              url: 'git@gitlab.59izt.com:kubernetes/spring-boot-project.git', 
              branch: env.gitlabBranch, 
              changelog: true, 
              poll: true, 
              credentialsId: 'gitlab-key'
            )
            script {
              COMMIT_ID = sh(
                returnStdout: true,
                script: "git log -n 1 --pretty=format:'%h'"
              ).trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }
          }
        }

      }
    }

    stage('Building') {
      steps {
        container(name: 'build') {
          sh '''
            curl repo.maven.apache.org 
            mvn clean install -DskipTests 
            ls target/*
          '''
        }

      }
    }

    stage('Docker build for creating image') {
      environment {
        HARBOR_USER = credentials('HARBOR_ACCOUNT')
      }
      steps {
        container(name: 'docker') {
          sh """
            echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
            docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
            docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
            docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
          """
        }
      }
    }

    stage('Deploying to K8s') {
      environment {
        MY_KUBECONFIG = credentials('kubernetes-test')
      }
      steps {
        container(name: 'kubectl') {
          sh """
            /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG \
              set image deploy -l \
              app=${IMAGE_NAME} \
              ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} \
              -n $NAMESPACE
          """
        }
      }
    }

  }
  environment {
    COMMIT_ID = ''
    HARBOR_ADDRESS = 'harbor.china-snow.net'
    REGISTRY_DIR = 'kubernetes'
    IMAGE_NAME = 'spring-boot-project'
    NAMESPACE = 'kubernetes'
    TAG = ''
  }
  parameters {
    gitParameter(
      branch: '',
      branchFilter: 'origin/(.*)', 
      defaultValue: '', 
      description: 'Branch for build and deploy', 
      name: 'BRANCH', 
      quickFilterEnabled: false, 
      selectedValue: 'NONE', 
      sortMode: 'NONE', 
      tagFilter: '*', 
      type: 'PT_BRANCH'
    )
  }
}

讲解 Jenkinsfile

首先是顶层的 Agent，定义的是 Kubernetes 的 Pod 作为 Jenkins 的 Slave

    agent {
        # 定义使用 kubernetes 作为 Agent
        kubernetes {
            # 选择的云为之前 jenkins 中配置的 kubernetes 集群名字
            cloud 'kubernetes-test'
            slaveConnectTimeout 1200
            # 将 workspace 改成 hostPath，因为该 slave 会在固定节点创建，如果有存储可用，可以改成 pvc 模式
            workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
            yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
  # jnlp 容器，主要负责和 Jenkins 的主节点通信
  - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
    image: 'registry.cn-beijing.aliyuncs.com/citools/jnlp:alpine'
    name: jnlp
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
  # build 容器，主要负责执行构建的命令，比如 Java 的需要 mvn 构建，就可以用一个 maven 的镜像
  - command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    # 使用 Maven 镜像，包含 mvn 工具。NodeJS 可以用 node 的镜像
    image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"
    imagePullPolicy: "IfNotPresent"
    name: "build"     # 容器的名字，流水线的 stage 可以直接使用该名字
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
    # Pod 单独创建了一个缓存的 volume，将其挂载到了 maven 插件的缓存目录，默认是 /root/.m2 目录
    - mountPath: "/root/.m2/"
      name: "cachedir"
      readOnly: false
  # 发版容器，因为最终是发布至 kubernetes 集群的，所以需要一个 kubectl 命令
  - command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    # 镜像的版本可以替换成其他的版本，也可以不进行替换，因为只执行 set 命令，所以版本是兼容的
    image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
    imagePullPolicy: "IfNotPresent"
    name: "kubectl"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
  # 用于生成镜像的容器，需要包含 docker 命令
  - command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
    imagePullPolicy: "IfNotPresent"
    name: "docker"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
    # 由于容器没有启动 docker 服务，所以将宿主机的 docker 经常挂载至容器即可
    - mountPath: "/var/run/docker.sock"
      name: "dockersock"
      readOnly: false
  restartPolicy: "Never"
  # 固定节点部署
  nodeSelector:
      build: "true"
  securityContext: {}
  volumes:
  # Docker socket volume
  - hostPath:
      path: "/var/run/docker.sock"
    name: "dockersock"
  - hostPath:
      path: "/usr/share/zoneinfo/Asia/Shanghai"
    name: "localtime"
  # maven 缓存目录
  - name: "cachedir" 
    hostPath:
      path: "/opt/m2"
'''    }
    }

之后看一下 Jenkinsfile 最后的环境变量和 parameters 的配置:

environment {
    COMMIT_ID = ""
    HARBOR_ADDRESS = "harbor.china-snow.net"    # Harbor 的地址
    REGISTRY_DIR = "kubernetes"                 # Harbor 的项目目录
    IMAGE_NAME = "spring-boot-project"          # 镜像的名称
    NAMESPACE = "kubernetes"                    # 该应用在 kubernetes 中的命名空间
    TAG = ""                                    # 镜像的 Tag，在此用 BUILD_TAG+COMMIT_ID 组成
}
parameters {
    # 之前讲过一些 choice、input 类型的参数，本次使用的是 GitParameter 插件
    # 该字段会在 Jenkins 页面生成一个选择分支的选项
    gitParameter(
        branch: '', 
        branchFilter: 'origin/(.*)', 
        defaultValue: '', 
        description: 'Branch for build and deploy', 
        name: 'BRANCH', 
        quickFilterEnabled: false, 
        selectedValue: 'NONE', 
        sortMode: 'NONE', 
        tagFilter: '*', 
        type: 'PT_BRANCH'
    )
}

接下来是拉代码的 stage，这个 stage 是一个并行的 stage，因为考虑了该流水线是手动触发还是自动触发:

stage('Pulling Code') { 
    parallel {
        stage('Pulling Code by Jenkins') {
            when {
                expression {
                    # 假如 env.gitlabBranch 为空，则该流水线为手动触发，那么就会执行该 stage，否则会执行同级的另外一个 stage
                    env.gitlabBranch == null
                }
            }
            steps {
                # 这里使用的是 git 插件拉取代码，BRANCH 变量取自于前面介绍的 parameters 配置
                git(
                    # git@gitlab.59izt.com:kubernetes/spring-boot-project.git 代码地址
                    changelog: 'git@gitlab.59izt.com:kubernetes/spring-boot-project.git',
                    branch: "${BRANCH}",
                    # 之前 Jenkins 上创建的拉取代码的 key
                    credentialsId: 'gitlab-key')
                script {
                    # 定义一些变量用于生成镜像的 Tag
                    # 获取最近一次提交的 Commit ID
                    COMMIT_ID = sh(
                            returnStdout: true,
                            script: "git log -n 1 --pretty=format:'%h'"
                        ).trim()
                    # 赋值给 TAG 变量，后面的 docker build 可以取到该 TAG 的值
                    TAG = BUILD_TAG + '-' + COMMIT_ID
                    println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
                }
            }
        }
        stage('Pulling Code by trigger') {
            when {
                expression {
                    # 如果 env.gitlabBranch 不为空，说明该流水线是通过 webhook 触发，则此时执行该 stage，上述的 stage 不再执行。此时 BRANCH 变量为空
                    env.gitlabBranch != null
                }
            }
            steps {
                # 以下配置和上述一致，只是此时 branch: env.gitlabBranch 取的值为 env.gitlabBranch
                git(
                    url: 'git@gitlab.59izt.com:kubernetes/spring-boot-project.git',
                    branch: env.gitlabBranch,
                    changelog: true,
                    poll: true,
                    credentialsId: 'gitlab-key'
                )
                script {
                    COMMIT_ID = sh(
                            returnStdout: true,
                            script: "git log -n 1 --pretty=format:'%h'"
                        ).trim()
                    TAG = BUILD_TAG + '-' + COMMIT_ID
                    println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
                }
            }
        }
    } 
}

代码拉下来后，就可以执行构建命令，由于本次实验是 Java 示例，所以需要使用 mvn 命令进行构建:

stage('Building') {
    steps {
        # 使用 Pod 模板里面的 build 容器进行构建
        container(name: 'build') {
            sh """
                # 编译命令，需要根据自己项目的实际情况进行修改，可能会不一致
                mvn clean install -DskipTests
                # 构建完成后，一般情况下会在 target 目录下生成 jar 包
                ls target/*
            """
        }
    }
}

生成编译产物后，需要根据该产物生成对应的镜像，此时可以使用 Pod 模板的 docker 容器:

stage('Docker build for creating image') {
    # 首先取出来 HARBOR 的账号密码
    environment {
        HARBOR_USER = credentials('HARBOR_ACCOUNT')
    }
    steps {
        # 指定使用 docker 容器
        container(name: 'docker') {
            sh """
                # 执行 build 命令，Dockerfile 会在下一小节创建，也是放在代码仓库，和 Jenkinsfile 同级
                echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
                docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
                # 登录 Harbor，HARBOR_USER_USR 和 HARBOR_USER_PSW 由上述 environment 生成
                docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
                # 将镜像推送至镜像仓库
                docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
            """
        }
    }
}

最后一步就是将该镜像发版至 Kubernetes 集群中，此时使用的是包含 kubectl 命令的容器:

stage('Deploying to K8s') {
    # 获取连接 Kubernetes 集群证书
    environment {
        MY_KUBECONFIG = credentials('kubernetes-test')
    }
    steps {
        # 指定使用 kubectl 容器
        container(name: 'kubectl'){
            sh """
                # 直接 set 更改 Deployment 的镜像即可
                /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
            """
        }
    }
}

注意：

本次发版的命令为 /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}: ${TAG} -n $NAMESPACE。由于本示例的 Deployment 名称、Label、容器名称均和项目名称一致，所以就统一了 IMAGE_NAME，如果示例不是按照此规范命名，需要进行更改。

定义 Dockerfile

在执行流水线过程时，需要将代码的编译产物做成镜像。Dockerfile 主要写的是如何生成公司业务的镜像。而本次示例是 Java 项目，只需要把 Jar 包放在有 Jre 环境的镜像中，然后启动该 Jar 包即可:

首先点击代码首页的“+”号，然后点击 New file，输入文件名为 Dockerfile，输入内容如下:

# 基础镜像可以按需修改，可以更改为公司自有镜像
FROM registry.cn-beijing.aliyuncs.com/dotbalo/jre:8u211-data
# jar 包名称改成实际的名称，本示例为 spring-cloud-eureka-0.0.1-SNAPSHOT.jar
COPY target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar ./
# 启动Jar包
CMD java -jar spring-cloud-eureka-0.0.1-SNAPSHOT.jar

定义 Kubernetes 资源

本示例在 GitLab 创建的 Group 为 kubernetes，可以将其认为是一个项目，同一个项目可以部署至 Kubernetes 集群中同一个 Namespace 中，本示例为 kubernetes 命名空间。由于使用的是私有仓库，因此也需要先配置拉取私有仓库镜像的密钥:

创建命名空间以及拉取镜像的 Secret

# 创建 Namespace
# kubectl create ns kubernetes
namespace/kubernetes created

# 创建 docker 镜像仓库使用的 secret
kubectl create secret docker-registry harborkey \
  --docker-server=harbor.china-snow.net \
  --docker-username=admin \
  --docker-password=Harbor12345 \
  --docker-email=2350686113@qq.com \
  -n kubernetes

配置该应用的 Deployment，service, ingress，创建 java-demo.yaml 文件，内容如下:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: spring-boot-project
  name: spring-boot-project
  namespace: kubernetes
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: spring-boot-project
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: spring-boot-project
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: app
                operator: In
                values:
                - spring-boot-project
      containers:
      - image: nginx
        imagePullPolicy: IfNotPresent
        name: spring-boot-project
        ports:
        - containerPort: 8761
          name: http
          protocol: TCP
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 1Gi
        lifecycle: {}
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8761
          timeoutSeconds: 2
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8761
          timeoutSeconds: 2
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: harborkey
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: spring-boot-project
  name: spring-boot-project
  namespace: kubernetes
spec:
  ports: # 端口按照实际情况进行修改
  - name: http
    port: 8761
    protocol: TCP
    targetPort: 8761
  selector:
    app: spring-boot-project
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spring-boot-project
  namespace: kubernetes
spec:
  rules:
  - host: spring-boot-project.test.com
    http: 
      paths:
      - backend:
          service:
            name: spring-boot-project
            port:
              number: 8761
        path: /
        pathType: ImplementationSpecific

注意：以上 Deployment 配置中配置了节点亲和性，所以最好在 k8s-node-03 上设置 label; 并且镜像初期可以使用其他镜像代理，比如 nginx

1 2	# 设置节点标签 kubectl label node k8s-node-03 app=spring-boot-project

创建 java-demo 资源

# kubectl create -f java-demo.yaml 
deployment.apps/spring-boot-project created
service/spring-boot-project created
ingress.networking.k8s.io/spring-boot-project created

在 Harbor 仓库中创建 kubernetes 项目
修改 jnpl 工作目录的权限，否则会无法启动 jnlp 容器
1
chmod 777 -R /opt/workspace

创建 Jenkins Job

登录 Jenkins –> 选择 New Item 输入 Job 名称，选择 Job 类型为 Pipeline
点击 OK 后，选择 Pipeline –> 选择 Pipeline script from SCM，SCM 选择 Git，输入 GitLab 上的项目地址，选择 Credentials 证书;
点击确定，有于 Pipeline 的特性，需要手动运行一次，才会出现 Build with Parameters 选项; 所以需要搜东点击一次 Build Now;
再次选择 spring-boot-project 项目，点击 Build with Parameters，会自动获取该项目所有的分支，选择一个分支，点击 Build，进行构建
可以点击 Bule Ocean 进入控制台查看构建状态，注意查看最后的镜像 TAG
查看 Harbor 仓库是否已经存在镜像

在 kubernetes 集群查看服务是否更新，查看镜像TAG是否是最新的镜像

# kubectl get pods -n kubernetes spring-boot-project-79d9496d65-99qkx -oyaml | grep image
    image: harbor.china-snow.net/kubernetes/spring-boot-project:jenkins-spring-boot-project-3-f769fb4
    imagePullPolicy: IfNotPresent
  imagePullSecrets:
    image: harbor.china-snow.net/kubernetes/spring-boot-project:jenkins-spring-boot-project-3-f769fb4
    imageID: harbor.china-snow.net/kubernetes/spring-boot-project@sha256:fe587363a363865418db501f0164d08c0bafe48c0a3a622630d143a9588f52e7

注意事项

如果 k8s slave Pod 创建失败并提示如下信息:

# kubectl logs -f spring-boot-project-4-2l1f5-x30lt-n36vv -c jnlp  
Exception in thread "main" java.io.IOException: The specified working directory should be fully accessible to the remoting executable (RWX): /home/jenkins/agent
        at org.jenkinsci.remoting.engine.WorkDirManager.verifyDirectory(WorkDirManager.java:249)
        at org.jenkinsci.remoting.engine.WorkDirManager.initializeWorkDir(WorkDirManager.java:202)
        at hudson.remoting.Engine.startEngine(Engine.java:251)
        at hudson.remoting.Engine.startEngine(Engine.java:227)
        at hudson.remoting.jnlp.Main.main(Main.java:228)
        at hudson.remoting.jnlp.Main._main(Main.java:223)
        at hudson.remoting.jnlp.Main.main(Main.java:189)

这是因为 jnlp 的工作目录权限不足，需要在 k8s 集群中用来创建 slave pod 的节点上对 /opt/workspace 目录进行授权

1	chmod 777 -R /opt/workspace/

一定要确保 Jenkins 中的 Credentials 凭证配置没问题，Harbor 需要提前创建好项目 kubernetes; 该装的插件要装齐;
如果使用域名访问 Harbor 以及 gitlab，一定要确保域名可以访问，或者使用 hosts 做本地解析;