kubernetes 更换证书

2023-10-18

字数统计: 2.4k字 | 阅读时长≈ 12分

参考文档

kubernetes 证书说明

Kubernetes 中使用了大量的证书，本文不会试图覆盖到所有可能使用到的证书，但会讨论到主要的证书。理解了这些证书的使用方法和原理后，也能很快理解其他可能遇到的证书文件。二进制 kubernetes 中各个组件证书的默认过期时间为: 根证书 ca.pem(10年)，其他的客户端证书为 1年

查看证书的到期时间

1	openssl x509 -noout -text -in /etc/kubernetes/ssl/ca.pem \|grep Not

生成组件证书

在 Kubeadm 安装方式下，初始化时会自动生成证书，但是在二进制安装方式下，需要手动生成证书，可以使用 OpenSSL 或者 cfssl。cfssl 是一个开源的证书管理工具，使用 json 文件生成证书，相比 openssl 更方便使用。

准备 cfssl 证书生成工具

找任意一台服务器操作，这里选择 k8s-master01 节点：

# 创建证书临时目录
mkdir cfssl && cd cfssl

# 下载 cfssl 工具
wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64

# 添加执行权限
chmod +x cfssljson_1.5.0_linux_amd64 cfssl_1.5.0_linux_amd64

# 将命令移动到可执行路劲并重命名
mv cfssl_1.5.0_linux_amd64 /usr/local/bin/cfssl
mv cfssljson_1.5.0_linux_amd64 /usr/local/bin/cfssljson

创建 CA 根证书和秘钥

CA (Certificate Authority) 是自签名的根证书，用来签名后续创建的其它证书。CA 证书是集群所有节点共享的，只需要创建一次，后续用它签名其它所有证书(除了etcd)

新建 CA 全局配置 ca-config.json 文件（后续各个组件生成证书时会用到该文件）

cat > ca-config.json<<EOF
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

新建 CA 的证书签名请求文件 ca-csr.json

cat > ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

生成 CA 证书和私钥
1
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/ssl/ca
以上命令会生成三个文件: ca.csr, ca-key.pem, ca.pem

创建 kube-apiserver 组件证书

创建 apiserver 证书签名请求文件 apiserver-csr.json

cat > apiserver-csr.json <<EOF
{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

创建 kube-apiserver 的证书

cfssl gencert \
-ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=ca-config.json \
-hostname=10.96.0.1,\
192.168.200.210,\
127.0.0.1,\
kubernetes,\
kubernetes.default,\
kubernetes.default.svc,\
kubernetes.default.svc.cluster,\
kubernetes.default.svc.cluster.local \
-profile=kubernetes \
apiserver-csr.json | cfssljson -bare /etc/kubernetes/ssl/apiserver

创建集群管理员 admin 的证书

创建 admin 证书签名请求文件 admin-csr.json

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

创建 admin 证书

cfssl gencert \
  -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare /etc/kubernetes/ssl/admin

创建 kubectl 使用的 admin kubeconfig 文件

export APISERVER=192.168.200.210:8443

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://${APISERVER} \
  --kubeconfig=/etc/kubernetes/cfg/admin.kubeconfig

kubectl config set-credentials kubernetes-admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --client-key=/etc/kubernetes/ssl/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=/etc/kubernetes/cfg/admin.kubeconfig

kubectl config set-context kubernetes-admin@kubernetes \
  --cluster=kubernetes \
  --user=kubernetes-admin \
  --kubeconfig=/etc/kubernetes/cfg/admin.kubeconfig

kubectl config use-context kubernetes-admin@kubernetes \
  --kubeconfig=/etc/kubernetes/cfg/admin.kubeconfig

创建 kube-controller-manager 证书

创建 kube-controller-manager 的证书签名请求文件 controller-manager-csr.json

cat > controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

创建 kube-controller-manager 证书

cfssl gencert \
  -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  controller-manager-csr.json | cfssljson -bare /etc/kubernetes/ssl/controller-manager

创建 kube-controller-manager 使用的 kubeconfig 文件

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://${APISERVER} \
  --kubeconfig=/etc/kubernetes/cfg/controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=/etc/kubernetes/ssl/controller-manager.pem \
  --client-key=/etc/kubernetes/ssl/controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=/etc/kubernetes/cfg/controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=/etc/kubernetes/cfg/controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager@kubernetes \
  --kubeconfig=/etc/kubernetes/cfg/controller-manager.kubeconfig

创建 kube-scheduler 证书

创建 kube-scheduler 证书请求签名文件 scheduler-csr.json

cat > scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

创建 kube-scheduler 证书

cfssl gencert \
-ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
scheduler-csr.json | cfssljson -bare /etc/kubernetes/ssl/scheduler

创建 Scheduler 使用的 kubeconfig 文件

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://${APISERVER} \
  --kubeconfig=/etc/kubernetes/cfg/scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
  --client-certificate=/etc/kubernetes/ssl/scheduler.pem \
  --client-key=/etc/kubernetes/ssl/scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=/etc/kubernetes/cfg/scheduler.kubeconfig

kubectl config set-context system:kube-scheduler@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=/etc/kubernetes/cfg/scheduler.kubeconfig

kubectl config use-context system:kube-scheduler@kubernetes \
  --kubeconfig=/etc/kubernetes/cfg/scheduler.kubeconfig

创建 ServiceAccount 证书

创建 sa.key

1	openssl genrsa -out /etc/kubernetes/ssl/sa.key 2048

创建 sa.pub

1	openssl rsa -in /etc/kubernetes/ssl/sa.key -pubout -out /etc/kubernetes/ssl/sa.pub

创建前端代理 CA 证书与客户端证书（配置 api 聚合层）

创建前端代理 CA 的 CSR 文件 front-proxy-ca-csr.json

cat > front-proxy-ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF

创建前端代理客户端的 CSR 文件 front-proxy-client-csr.json

cat > front-proxy-client-csr.json <<EOF
{
  "CN": "front-proxy-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}
EOF

创建前端代理 CA 证书以及客户端证书

# 创建前端代理 CA 证书
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/ssl/front-proxy-ca

# 创建前端代理客户端证书
cfssl gencert \
  -ca=/etc/kubernetes/ssl/front-proxy-ca.pem \
  -ca-key=/etc/kubernetes/ssl/front-proxy-ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/ssl/front-proxy-client

创建 worker 节点证书

配置 kubelet 自动签发证书的 TLS Bootstrapping

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://${APISERVER} \
  --kubeconfig=/etc/kubernetes/cfg/bootstrap-kubelet.kubeconfig

kubectl config set-credentials tls-bootstrap-token-user \
  --token=c8ad9c.2e4d610cf3e7426e \
  --kubeconfig=/etc/kubernetes/cfg/bootstrap-kubelet.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes \
  --cluster=kubernetes \
  --user=tls-bootstrap-token-user \
  --kubeconfig=/etc/kubernetes/cfg/bootstrap-kubelet.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
  --kubeconfig=/etc/kubernetes/cfg/bootstrap-kubelet.kubeconfig

注意：如果要修改 bootstrap.secret.yaml 的 token-id 和 token-secret，需要保证下图红圈内的字符串一致的，并且位数是一样的。还要保证上个命令的 token：c8ad9c.2e4d610cf3e7426e 与你修改的字符串要一致

创建 kube-proxy 使用的 kubeconfig 文件

# 创建 kube-proxy 访问集群的 sa 账户
kubectl create serviceaccount -n kube-system kube-proxy

# 创建 clusterrolebinding
kubectl create clusterrolebinding system:kube-proxy \
  --clusterrole system:node-proxier \
  --serviceaccount kube-system:kube-proxy

# 
SECRET=$(kubectl get sa -n kube-system kube-proxy \
  --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl get secret -n kube-system ${SECRET} \
  --output=jsonpath='{.data.token}' | base64 -d)

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://${APISERVER} \
  --kubeconfig=/etc/kubernetes/cfg/kube-proxy.kubeconfig

kubectl config set-credentials kubernetes \
  --token=${JWT_TOKEN} \
  --kubeconfig=/etc/kubernetes/cfg/kube-proxy.kubeconfig

kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=kubernetes \
  --kubeconfig=/etc/kubernetes/cfg/kube-proxy.kubeconfig

kubectl config use-context kubernetes \
  --kubeconfig=/etc/kubernetes/cfg/kube-proxy.kubeconfig

创建 kubelet 的证书自动签发配置 (可选)

创建 bootstrap.secret.yaml 文件，内容如下

cat > bootstrap.secert.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c
  token-secret: 2e4d610cf3e7426e
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-certificate-rotation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver
EOF

创建资源

1	kubectl create -f bootstrap.secret.yaml