kubeadm 高可用 K8S 集群安装(1.36.x)

集群环境说明

软件版本

类型	名称
系统版本	Ubuntu 26.04.1 LTS
kubernetes 版本	1.36.x
部署方式	kubeadm
网络插件	cilium
高可用架构	keepalived + haproxy
Dashboard	headlamp-v0.46.0

服务器配置

序号	主机名	IP地址	角色	备注
1	k8smaster01	192.168.0.31	master	kubeadm、kubelet、kubectl、Keepalived、haproxy、etcd
2	k8smaster02	192.168.0.32	master	kubeadm、kubelet、kubectl、Keepalived、haproxy、etcd
3	k8smaster03	192.168.0.33	master	kubeadm、kubelet、kubectl、Keepalived、haproxy、etcd
4	k8snode01	192.168.0.34	node	kubeadm、kubelet
5	k8snode02	192.168.0.35	node	kubeadm、kubelet
6	k8snode03	192.168.0.36	node	kubeadm、kubelet
7	cluster-endpoint	192.168.0.30	VIP	这是集群 VIP 映射的主机名，不是具体的服务器

环境初始化配置

1. 关闭终端一直提示的 08;end=xxx;exit=xxxx;start=xxxxxx;machineid=xxxxx;user=xxx;hostname=xxxx;bootid=xxxx;pid=xxxxxx;type=xxxxx;cwd=xxxxx 提示

   mv /usr/lib/systemd/profile.d/80-systemd-osc-context.sh /usr/lib/systemd/profile.d/80-systemd-osc-context.sh.bak
   ln -s /dev/null /etc/tmpfiles.d/20-systemd-osc-context.conf
   
   重启
   reboot

2. 精简登录时服务器信息提示

   cd /etc/update-motd.d/ && chmod -x * && chmod +x 00-header

3. 所有节点修改主机名

   # 192.168.0.31
   hostnamectl set-hostname k8smaster01
   
   # 192.168.0.32
   hostnamectl set-hostname k8smaster02
   
   # 192.168.0.33
   hostnamectl set-hostname k8smaster03
   
   # 192.168.0.34
   hostnamectl set-hostname k8snode01
   
   # 192.168.0.35
   hostnamectl set-hostname k8snode02
   
   # 192.168.0.36
   hostnamectl set-hostname k8snode03

4. 所有节点配置 hosts 解析

   cat >> /etc/hosts <<EOF
   192.168.0.31    k8smaster01
   192.168.0.32    k8smaster02
   192.168.0.33    k8smaster03
   192.168.0.34    k8snode01
   192.168.0.35    k8snode02
   192.168.0.36    k8snode03
   192.168.0.30    cluster-endpoint
   EOF

5. 所有节点修改 APT 软件源为阿里云

   # 备份文件
   cp /etc/apt/sources.list{,.bak}
   cp /etc/apt/sources.list.d/ubuntu.sources{,.bak}
   
   # 修改源
   cat > /etc/apt/sources.list.d/ubuntu.sources <<'EOF'
   Types: deb
   URIs: https://mirrors.aliyun.com/ubuntu/
   Suites: resolute resolute-updates resolute-backports
   Components: main restricted universe multiverse
   Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
   
   Types: deb
   URIs: https://mirrors.aliyun.com/ubuntu/
   Suites: resolute-security
   Components: main restricted universe multiverse
   Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
   EOF
   
   # 更新缓存
   apt clean && apt update && apt upgrade

6. 所有节点安装常用的软件

   apt install -y vim wget curl psmisc bash-completion net-tools network-manager lrzsz unzip jq

7. 所有节点配置时间同步以及时区

   # 注释掉原有的时间池
   sed -i '/^pool .* prefer$/s/^/# /' /etc/chrony/sources.d/ubuntu-ntp-pools.sources 
   
   # 新增时间池
   echo 'pool ntp.aliyun.com iburst maxsources 1 nts prefer' >> /etc/chrony/sources.d/ubuntu-ntp-pools.sources 
   
   # 重启 chronyd 服务
   systemctl restart chronyd
   
   # 同步时间
   chronyc sources -v
   
   # 修改时区
   timedatectl set-timezone Asia/Shanghai

8. 所有节点修改 PS1 显示样式

   [ -z "$(grep ^PS1 ~/.bashrc)" ] && echo "PS1='\${debian_chroot:+(\$debian_chroot)}\\[\\e[1;32m\\]\\u@\\h\\[\\033[00m\\]:\\[\\033[01;34m\\]\\w\\[\\033[00m\\]\\$ '" >> ~/.bashrc

9. 所有节点修改历史命令记录配置

   cat >> /etc/profile << 'EOF'
   
   # 获取登录 IP
   USER_IP=$(who -u am i 2>/dev/null | awk '{print $NF}' | sed -e 's/[()]//g')
   if [ -z "$USER_IP" ]; then
       USER_IP=$(hostname)
   fi
   
   # 历史格式：时间 IP 用户名
   export HISTTIMEFORMAT="%F %T $USER_IP $(whoami) "
   
   # 实时写入历史文件
   export PROMPT_COMMAND="history -a; $PROMPT_COMMAND"
   EOF

10. 所有节点卸载 cloud-init 以及卸载自动更新工具

    # 删除云功能和自动升级功能
    sudo apt remove -y unattended-upgrades cloud-init
    
    # 清理相关依赖
    sudo apt install -f && sudo apt autoremove

11. 所有节点关闭防火墙 ufw

    systemctl disable --now ufw

12. 所有节点修改 SSH 配置，允许 root 用户登录

    sed -i 's/#PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config
    
    systemctl restart ssh.service
    systemctl restart ssh.socket 

13. 所有节点关闭交换分区 swap

    sed -ri 's/.*swap.*/#&/' /etc/fstab
    swapoff -a && sysctl -w vm.swappiness=0

14. k8smaster01 配置免密登录其他节点

    # 生成秘钥
    ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ''
    
    # 复制公钥到其他节点
    for i in k8smaster01 k8smaster02 k8smaster03 k8snode01 k8snode02 k8snode03;do \
        ssh-copy-id -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no root@$i; \
    done

15. 安装配置 ipvs

    # 安装 ipvs 相关软件
    apt install -y ipvsadm ipset sysstat conntrack libseccomp-dev
    
    # 配置 ipvs
    cat > /etc/modules-load.d/ipvs.conf <<EOF
    ip_vs
    ip_vs_lc
    ip_vs_wlc
    ip_vs_rr
    ip_vs_wrr
    ip_vs_lblc
    ip_vs_lblcr
    ip_vs_dh
    ip_vs_sh
    ip_vs_fo
    ip_vs_nq
    ip_vs_sed
    ip_vs_ftp
    nf_conntrack
    ip_tables
    ip_set
    xt_set
    ipt_set
    ipt_rpfilter
    ipt_REJECT
    ipip
    EOF
    
    systemctl restart systemd-modules-load.service
    
    # 检查模块是否加载
    lsmod | grep -e ip_vs -e nf_conntrack

16. 所有节点修改内核参数

    cat > /etc/sysctl.d/k8s.conf << EOF
    net.ipv4.ip_forward = 1
    net.bridge.bridge-nf-call-iptables = 1
    fs.may_detach_mounts = 1
    vm.overcommit_memory=1
    vm.panic_on_oom=0
    fs.inotify.max_user_watches=89100
    fs.file-max=52706963
    fs.nr_open=52706963
    net.netfilter.nf_conntrack_max=2310720
    
    net.ipv4.tcp_keepalive_time = 600
    net.ipv4.tcp_keepalive_probes = 3
    net.ipv4.tcp_keepalive_intvl =15
    net.ipv4.tcp_max_tw_buckets = 36000
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_max_orphans = 327680
    net.ipv4.tcp_orphan_retries = 3
    net.ipv4.tcp_syncookies = 1
    net.ipv4.tcp_max_syn_backlog = 16384
    net.ipv4.ip_conntrack_max = 65536
    net.ipv4.tcp_max_syn_backlog = 16384
    net.ipv4.tcp_timestamps = 0
    net.core.somaxconn = 16384
    
    net.ipv6.conf.all.disable_ipv6 = 0
    net.ipv6.conf.default.disable_ipv6 = 0
    net.ipv6.conf.lo.disable_ipv6 = 0
    net.ipv6.conf.all.forwarding = 1
    EOF
    
    sysctl --system

17. 所有节点修改服务器 limits 限制

    # 临时修改
    ulimit -SHn 65535
    
    # 永久生效
    sed -i '/^# End/i\* soft nofile    655350' /etc/security/limits.conf
    sed -i '/^# End/i\* hard nofile    131072' /etc/security/limits.conf
    sed -i '/^# End/i\* soft nproc    655350' /etc/security/limits.conf
    sed -i '/^# End/i\* hard nproc    655350' /etc/security/limits.conf
    sed -i '/^# End/i\* soft memlock   unlimited' /etc/security/limits.conf
    sed -i '/^# End/i\* hard memlock   unlimited' /etc/security/limits.conf

安装基础组件

安装容器运行时 Containerd

1. 所有节点安装 Containerd

   # 卸载旧的的 docker
   for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
      
   # 配置 Docker apt 仓库
   sudo apt-get update
   sudo apt-get install -y ca-certificates curl
   sudo install -m 0755 -d /etc/apt/keyrings
   sudo wget https://download.docker.com/linux/ubuntu/gpg && sudo mv gpg /etc/apt/keyrings/docker.asc
   sudo chmod a+r /etc/apt/keyrings/docker.asc
   
   # Add the repository to Apt sources:
   echo \
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
     $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   
   sudo apt-get update
   
   # 安装 Containerd 
   sudo apt-get install containerd.io 

   containerd 与 kubernetes 版本兼容查询地址 Kubernetes Support

2. 所有节点配置 Containerd 所需的模块：

   cat << EOF | sudo tee /etc/modules-load.d/containerd.conf
   overlay
   br_netfilter
   EOF
   
   modprobe -- overlay
   modprobe -- br_netfilter

3. 所有节点配置 Containerd 的配置文件

   # 创建配置文件目录
   mkdir -p /etc/containerd
   
   # 生成默认配置文件
   containerd config default | tee /etc/containerd/config.toml
   
   # 修改 SystemdCgroup 配置
   sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
   cat /etc/containerd/config.toml | grep SystemdCgroup
   
   # 修改 pause 镜像
   sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/kubernetes-z#g" /etc/containerd/config.toml
   cat /etc/containerd/config.toml | grep sandbox

4. 所有节点启动Containerd，并配置开机自启动

   systemctl daemon-reload
   systemctl enable --now containerd

5. 所有节点安装配置 crictl 客户端

   # 下载 crictl
   wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.36.0/crictl-v1.36.0-linux-amd64.tar.gz
   
   # 安装 crictl
   tar xf crictl-v*-linux-amd64.tar.gz -C /usr/bin/
   
   # 生成 crictl 配置文件
   cat > /etc/crictl.yaml <<EOF
   runtime-endpoint: unix:///run/containerd/containerd.sock
   image-endpoint: unix:///run/containerd/containerd.sock
   timeout: 10
   debug: false
   EOF
   
   # 测试 crictl
   crictl info

安装 Keepalived + HAproxy

这里高可用采用的是 HAProxy + Keepalived，HAProxy 和 Keepalived 以守护进程的方式在所有 Master 节点部署。

1. 所有 Master 节点安装 HAProxy 和 Keepalived。

   apt install -y haproxy keepalived

2. 所有 Master 节点配置 HAProxy（详细配置参考 HAProxy 文档，所有 MAster 节点的 HAProxy 配置相同）

   mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy_ori.cfg
   
   cat > /etc/haproxy/haproxy.cfg <<EOF
   global
       log         127.0.0.1 local0 err
       chroot      /var/lib/haproxy
       pidfile     /var/run/haproxy.pid
       maxconn     4000
       ulimit-n    16384
       user        haproxy
       group       haproxy
       stats timeout   30s
       daemon
       # turn on stats unix socket
       stats socket /var/lib/haproxy/stats
   
   defaults
       mode                    http
       log                     global
       option                  httplog
       option                  dontlognull
       timeout http-request    15s
       timeout queue           1m
       timeout connect         5000
       timeout client          50000
       timeout server          50000
       timeout http-keep-alive 15s
       timeout check           15s
       maxconn                 3000
   
   frontend  monitor-in
       bind    *:33305
       mode    http
       option  httplog
       monitor-uri     /monitor
   
   listen stats
       bind    *:8006
       mode    http
       stats   enable
       stats   hide-version
       stats   uri     /stats
       stats   refresh 30s
       stats   realm   Haproxy\ Statistics
       stats   auth    admin:admin
   
   frontend k8s-master
       bind    0.0.0.0:8443
       bind    127.0.0.1:8443
       mode    tcp
       option  tcplog
       tcp-request inspect-delay   5s
       default_backend K8S-master
       
   backend static
       balance     roundrobin
       server      static 127.0.0.1:4331 check
   
   backend K8S-master
       mode    tcp
       option  tcplog
       option  tcp-check
       balance     roundrobin
       default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
       server  k8smaster01 192.168.0.31:6443 check
       server  k8smaster02 192.168.0.32:6443 check
       server  k8smaster03 192.168.0.33:6443 check
   EOF

3. 所有 Master 节点配置 Keepalived。注意修改 interface（服务网卡），priority（优先级，不同即可）mcast_src_ip（节点IP地址）。

   mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived_ori.conf
   
   export INTERFACE=$(ip route show |grep default |cut -d ' ' -f5)
   export IPADDR=$(ifconfig |grep -A1 $INTERFACE |grep inet |awk '{print $2}')
   
   cat > /etc/keepalived/keepalived.conf <<EOF
   ! Configuration File for keepalived
   global_defs {
       router_id LVS_DEVEL
   }
   
   vrrp_script chk_apiserver {
       script "/etc/keepalived/check_apiserver.sh"
       interval    2
       weight      -5
       fall    3
       rise    2
   }
   
   vrrp_instance VI_1 {
       state MASTER
       interface ${INTERFACE}
       mcast_src_ip ${IPADDR}
       virtual_router_id 30
       priority 100
       advert_int 2
       authentication {
           auth_type PASS
           auth_pass K8SHA_KA_AUTH
       }
       virtual_ipaddress {
           192.168.0.30
       }
       track_script {
           chk_apiserver
       }
   }
   EOF
   
   # 修改其他 master 节点配置文件的 priority 值
   # k8smaster02 节点改为 priority 99, state 改为 BACKUP
   # k8smaster03 节点改为 priority 98, state 改为 BACKUP

   如果一个局域网内有多个 k8s 集群，需要修改 virtual_router_id 的值为唯一的，可以设置为 VIP 地址最后的一个值，比如这里的 30

4. 所有 Master 节点配置 Keepalived 健康检查文件

   cat > /etc/keepalived/check_apiserver.sh <<EOF
   #!/bin/bash
   
   err=0
   for k in \$(seq 1 3); do
       check_code=\$(pgrep haproxy)
       if [ -z "\$check_code" ]; then
           err=\$((err + 1))
           sleep 1
       else
           err=0
           break
       fi
   done
   
   if [ \$err -ne 0 ]; then
       echo "Stopping keepalived service"
       sudo /usr/bin/systemctl stop keepalived
       exit 1
   else
       echo "HAProxy is running. Keepalived service remains active."
       exit 0
   fi
   EOF
   
   chmod +x /etc/keepalived/check_apiserver.sh

5. 所有 Master 节点启动 HAProxy 和 Keepalived

   systemctl enable --now haproxy.service keepalived.service

部署 kubernetes 集群

安装 kubernetes 基础组件

1. 所有节点更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包：

   sudo apt-get update
   sudo apt-get install -y apt-transport-https ca-certificates curl gpg

2. 下载用于 Kubernetes 软件包仓库的公共签名密钥。所有仓库都使用相同的签名密钥，因此你可以忽略URL中的版本：

   # 如果 `/etc/apt/keyrings` 目录不存在，则应在 curl 命令之前创建它，请阅读下面的注释。
   # sudo mkdir -p -m 755 /etc/apt/keyrings
   curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key | \
   sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

3. 添加 Kubernetes apt 仓库。 请注意，此仓库仅包含适用于 Kubernetes 1.36 的软件包； 对于其他 Kubernetes 次要版本，则需要更改 URL 中的 Kubernetes 次要版本以匹配你所需的次要版本

   # 此操作会覆盖 /etc/apt/sources.list.d/kubernetes.list 中现存的所有配置。
   echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

4. 更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并锁定其版本

   sudo apt-get update
   sudo apt-get install -y kubelet kubeadm kubectl
   sudo apt-mark hold kubelet kubeadm kubectl

   非 master 节点可以选择不安装 kubectl

5. （可选）先启用 kubelet 服务，再运行 kubeadm

   sudo systemctl enable --now kubelet

初始化安装 kubernetes

1. 查看新版本有哪些镜像

   kubeadm config images list --image-repository registry.cn-hangzhou.aliyuncs.com/kubernetes-z

2. 所有节点执行以下命令将镜像下载到本地

   # 只有一个CRI的情况下
   kubeadm config images pull --image-repository registry.cn-hangzhou.aliyuncs.com/kubernetes-z
   
   # 指定CRI拉去镜像
   # kubeadm config images pull --image-repository registry.cn-hangzhou.aliyuncs.com/kubernetes-z --cri-socket unix:///var/run/cri-dockerd.sock
   # kubeadm config images pull --image-repository registry.cn-hangzhou.aliyuncs.com/kubernetes-z --cri-socket unix:///var/run/containerd/containerd.sock

3. 修改 kubeadm 集群初始化的配置文件

   # 创建默认配置文件
   kubeadm config print init-defaults > kubeadm-init.yaml
   
   # 修改 kubeadm-init.yaml 文件，修改后文件内容如下
   apiVersion: kubeadm.k8s.io/v1beta4
   bootstrapTokens:
   - groups:
     - system:bootstrappers:kubeadm:default-node-token
     token: 7t2weq.bjbawausm0jaxury
     ttl: 24h0m0s
     usages:
     - signing
     - authentication
   kind: InitConfiguration
   localAPIEndpoint:
     advertiseAddress: 192.168.0.31
     bindPort: 6443
   nodeRegistration:
     criSocket: unix:///var/run/containerd/containerd.sock
     imagePullPolicy: IfNotPresent
     imagePullSerial: true
     name: k8smaster01
     taints:
     - effect: PreferNoSchedule
       key: node-role.kubernetes.io/master
     - effect: NoSchedule
       key: node-role.kubernetes.io/control-plane
     kubeletExtraArgs:
     - name: tls-cipher-suites
       value: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
     - name: cgroup-driver
       value: systemd
   timeouts:
     controlPlaneComponentHealthCheck: 4m0s
     discovery: 5m0s
     etcdAPICall: 2m0s
     kubeletHealthCheck: 4m0s
     kubernetesAPICall: 1m0s
     tlsBootstrap: 5m0s
     upgradeManifests: 5m0s
   ---
   apiServer:
     certSANs:
     - 127.0.0.1
     - 192.168.0.30
     - 192.168.0.31
     - 192.168.0.32
     - 192.168.0.33
     - k8smaster01
     - k8smaster02
     - k8smaster03
     - cluster-endpoint
   apiVersion: kubeadm.k8s.io/v1beta4
   caCertificateValidityPeriod: 87600h0m0s
   certificateValidityPeriod: 8760h0m0s
   certificatesDir: /etc/kubernetes/pki
   clusterName: kubernetes
   controllerManager: {}
   dns: {}
   encryptionAlgorithm: RSA-2048
   etcd:
     local:
       dataDir: /var/lib/etcd
   imageRepository: registry.cn-hangzhou.aliyuncs.com/kubernetes-z
   kind: ClusterConfiguration
   kubernetesVersion: 1.36.5
   networking:
     dnsDomain: cluster.local
     serviceSubnet: 10.96.0.0/16
     podSubnet: 172.16.0.0/16
   proxy: {}
   scheduler: {}
   controlPlaneEndpoint: "cluster-endpoint:8443"
   ---
   apiVersion: kubeproxy.config.k8s.io/v1alpha1
   kind: KubeProxyConfiguration
   mode: ipvs
   ---
   apiVersion: kubelet.config.k8s.io/v1beta1
   kind: KubeletConfiguration
   address: 0.0.0.0
   port: 10250
   readOnlyPort: 0
   authentication:
     anonymous:
       enabled: false
     webhook:
       cacheTTL: 2m0s
       enabled: true
     x509:
       clientCAFile: /etc/kubernetes/pki/ca.crt
   authorization:
     mode: Webhook
     webhook:
       cacheAuthorizedTTL: 5m0s
       cacheUnauthorizedTTL: 30s
   cgroupDriver: systemd
   cgroupsPerQOS: true
   clusterDNS:
   - 10.96.0.10
   clusterDomain: cluster.local
   containerLogMaxFiles: 5
   containerLogMaxSize: 10Mi
   contentType: application/vnd.kubernetes.protobuf
   cpuCFSQuota: true
   cpuManagerPolicy: none
   cpuManagerReconcilePeriod: 10s
   enableControllerAttachDetach: true
   enableDebuggingHandlers: true
   enforceNodeAllocatable:
   - pods
   eventBurst: 10
   eventRecordQPS: 5
   evictionHard:
     imagefs.available: 15%
     memory.available: 100Mi
     nodefs.available: 10%
     nodefs.inodesFree: 5%
   evictionPressureTransitionPeriod: 5m0s
   failSwapOn: true
   fileCheckFrequency: 20s
   hairpinMode: promiscuous-bridge
   healthzBindAddress: 127.0.0.1
   healthzPort: 10248
   httpCheckFrequency: 20s
   imageGCHighThresholdPercent: 85
   imageGCLowThresholdPercent: 80
   imageMinimumGCAge: 2m0s
   iptablesDropBit: 15
   iptablesMasqueradeBit: 14
   kubeAPIBurst: 10
   kubeAPIQPS: 5
   makeIPTablesUtilChains: true
   maxOpenFiles: 1000000
   maxPods: 110
   nodeStatusUpdateFrequency: 10s
   oomScoreAdj: -999
   podPidsLimit: -1
   registryBurst: 10
   registryPullQPS: 5
   resolvConf: /run/systemd/resolve/resolv.conf
   rotateCertificates: true
   runtimeRequestTimeout: 2m0s
   serializeImagePulls: true
   staticPodPath: /etc/kubernetes/manifests
   streamingConnectionIdleTimeout: 4h0m0s
   syncFrequency: 1m0s
   volumeStatsAggPeriod: 1m0s
   serverTLSBootstrap: true
   rotateServerCertificates: true
   kubeReserved:
     cpu: "0.3"
     memory: 200Mi
     ephemeral-storage: 1Gi
   systemReserved:
     cpu: "0.3"
     memory: 200Mi
     ephemeral-storage: 1Gi

4. 开始初始化集群

   kubeadm init --config kubeadm-init.yaml --upload-certs
   
   [init] Using Kubernetes version: v1.36.5
   [preflight] Running pre-flight checks
   [preflight] Pulling images required for setting up a Kubernetes cluster
   [preflight] This might take a minute or two, depending on the speed of your internet connection
   [preflight] You can also perform this action beforehand using 'kubeadm config images pull'
   [certs] Using certificateDir folder "/etc/kubernetes/pki"
   [certs] Generating "ca" certificate and key
   [certs] Generating "apiserver" certificate and key
   
   ... 
   
   Your Kubernetes control-plane has initialized successfully!
   
   To start using your cluster, you need to run the following as a regular user:
   
   mkdir -p $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config
   
   Alternatively, if you are the root user, you can run:
   
   export KUBECONFIG=/etc/kubernetes/admin.conf
   
   You should now deploy a pod network to the cluster.
   Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
   https://kubernetes.io/docs/concepts/cluster-administration/addons/
   
   You can now join any number of control-plane nodes running the following command on each as root:
   
   kubeadm join cluster-endpoint:8443 --token 7t2weq.bjbawausm0jaxury \
           --discovery-token-ca-cert-hash sha256:1dfd1af04a89eb680b6b555bdc0b5e3580933745de993923d93d9ece03be6fc3 \
           --control-plane --certificate-key 15e04b41a28176e45336d4f1aa6b0c1fa14a56e0e05302c41804f93987f0956b
   
   Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
   As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
   "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
   
   Then you can join any number of worker nodes by running the following on each as root:
   
   kubeadm join cluster-endpoint:8443 --token 7t2weq.bjbawausm0jaxury \
           --discovery-token-ca-cert-hash sha256:1dfd1af04a89eb680b6b555bdc0b5e3580933745de993923d93d9ece03be6fc3 

   初始化时加 --upload-certs 参数，这样其他 master 加入时才能自动拉取证书。

   初始化失败的话清理重置操作：

   kubeadm reset -f --cri-socket=unix:///var/run/containerd/containerd.sock
   rm -rf /etc/kubernetes/ /var/lib/kubelet/ /var/lib/etcd/
   systemctl restart containerd kubelet

   kubeadm init 超时后，只要控制平面已经起来，就不需要重新 init，直接在 master 上重新生成 join 命令即可。分两种情况：

   # 生成 Worker 节点加入命令
   kubeadm token create --print-join-command
   
   # 生成 Master 节点加入命令
   echo "$(kubeadm token create --print-join-command) \
     --control-plane --certificate-key \
     $(kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)"

5. 配置 kubectl

   mkdir -p $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config

6. 其他节点节点加入集群后，查看集群状态

   # kubectl get nodes
   NAME          STATUS     ROLES           AGE     VERSION
   k8smaster01   NotReady   control-plane   102m    v1.36.5
   k8smaster02   NotReady   control-plane   92m     v1.36.5
   k8smaster03   NotReady   control-plane   2m26s   v1.36.5
   k8snode01     NotReady   <none>          22s     v1.36.5
   k8snode02     NotReady   <none>          15s     v1.36.5
   k8snode03     NotReady   <none>          12s     v1.36.5

安装网络插件 cilium

1. 安装 Helm

   wget https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
   tar -xf helm-*-linux-amd64.tar.gz 
   mv linux-amd64/helm /usr/local/bin/

2. 安装 cilium

   # 添加源
   helm repo add cilium https://helm.cilium.io
   
   # 修改镜像使用国内仓库
   helm pull cilium/cilium
   tar xvf cilium-*.tgz
   cd cilium/
   
   sed -i "s#quay.io/#quay.m.daocloud.io/#g" values.yaml
   
   # 默认安装参数
   # helm install  cilium ./cilium/ -n kube-system
   
   # 启用ipv6
   # helm install cilium ./cilium/ --namespace kube-system --set ipv6.enabled=true
   
   # 启用路由信息和监控插件
   helm install cilium ./cilium/ \
     --namespace kube-system \
     --set ipv6.enabled=true \
     --set hubble.relay.enabled=true \
     --set hubble.ui.enabled=true \
     --set prometheus.enabled=true \
     --set operator.prometheus.enabled=true \
     --set hubble.enabled=true \
     --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" 

3. 查看安装状态

   kubectl  get pod -A | grep cil

4. 部署专属监控面板

   # 下载资源清单文件
   wget https://github.com/cilium/cilium/blob/main/examples/kubernetes/addons/prometheus/monitoring-example.yaml
   
   # 修改 grafana 镜像仓库地址
   sed -i "s#docker.io/#registry.cn-hangzhou.aliyuncs.com/59izt-publics/#g" monitoring-example.yaml
   
   # 修改 Prometheus 镜像仓库地址
   sed -i "s#prom/#registry.cn-hangzhou.aliyuncs.com/59izt-publics/#g" monitoring-example.yaml
   
   # 安装监控面板
   kubectl  apply -f monitoring-example.yaml

5. 访问 Grafana 以及 Prometheus

   # 暴露 Grafana 服务端口
   kubectl -n cilium-monitoring port-forward service/grafana --address 0.0.0.0 --address :: 3000:3000
   
   # 暴露 Prometheus 服务端口
   kubectl -n cilium-monitoring port-forward service/prometheus --address 0.0.0.0 --address :: 9090:9090
   
   # 暴露 hubble-ui 服务端口
   kubectl -n kube-system port-forward services/hubble-ui --address 0.0.0.0 --address :: 8081:80

6. 查看集群状态

   # kubectl get nodes  
   NAME          STATUS   ROLES           AGE     VERSION
   k8smaster01   Ready    control-plane   5h43m   v1.36.5
   k8smaster02   Ready    control-plane   5h33m   v1.36.5
   k8smaster03   Ready    control-plane   4h3m    v1.36.5
   k8snode01     Ready    <none>          4h1m    v1.36.5
   k8snode02     Ready    <none>          4h      v1.36.5
   k8snode03     Ready    <none>          4h      v1.36.5
   
   # kubectl get pods -n kube-system 
   NAME                                  READY   STATUS    RESTARTS       AGE
   cilium-8hwd2                          1/1     Running   0              68m
   cilium-b9grx                          1/1     Running   0              68m
   cilium-bv2p2                          1/1     Running   0              68m
   cilium-dbr58                          1/1     Running   0              68m
   cilium-envoy-9zxr2                    1/1     Running   0              68m
   cilium-envoy-dnwwm                    1/1     Running   0              68m
   cilium-envoy-nsgr7                    1/1     Running   0              68m
   cilium-envoy-v5976                    1/1     Running   0              68m
   cilium-envoy-vkk4m                    1/1     Running   0              68m
   cilium-envoy-zsgx9                    1/1     Running   0              68m
   cilium-jbwvw                          1/1     Running   0              68m
   cilium-jfzzc                          1/1     Running   0              68m
   cilium-operator-8c795fff6-vvjbh       1/1     Running   0              68m
   cilium-operator-8c795fff6-xs7rh       1/1     Running   0              68m
   coredns-7dd4c9d98d-57qff              1/1     Running   0              5h43m
   coredns-7dd4c9d98d-gqkw6              1/1     Running   0              5h43m
   etcd-k8smaster01                      1/1     Running   4 (4h8m ago)   5h43m
   etcd-k8smaster02                      1/1     Running   2 (4h8m ago)   5h33m
   etcd-k8smaster03                      1/1     Running   0              4h3m
   hubble-relay-54ffc877b8-r5pg6         1/1     Running   0              68m
   hubble-ui-74ff94bbbb-czcsr            2/2     Running   0              68m
   kube-apiserver-k8smaster01            1/1     Running   4 (4h8m ago)   5h43m
   kube-apiserver-k8smaster02            1/1     Running   2 (4h8m ago)   5h33m
   kube-apiserver-k8smaster03            1/1     Running   0              4h3m
   kube-controller-manager-k8smaster01   1/1     Running   4 (4h8m ago)   5h43m
   kube-controller-manager-k8smaster02   1/1     Running   2 (4h8m ago)   5h33m
   kube-controller-manager-k8smaster03   1/1     Running   0              4h3m
   kube-proxy-c4b95                      1/1     Running   0              4h1m
   kube-proxy-kvpld                      1/1     Running   2 (4h8m ago)   5h33m
   kube-proxy-p69q2                      1/1     Running   0              4h3m
   kube-proxy-vwlb7                      1/1     Running   2 (4h8m ago)   5h43m
   kube-proxy-xftm4                      1/1     Running   0              4h1m
   kube-proxy-zxqkl                      1/1     Running   0              4h1m
   kube-scheduler-k8smaster01            1/1     Running   4 (4h8m ago)   5h43m
   kube-scheduler-k8smaster02            1/1     Running   2 (4h8m ago)   5h33m
   kube-scheduler-k8smaster03            1/1     Running   0              4h3m

安装 Metrics-Server

1. 下载 metrics-server 资源清单

   wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

2. 修改 metrics-server 文件，主要修改内容如下:

   spec:
     # 0. 添加节点亲和性，将 metrics 服务部署到非边缘节点
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: node-role.kubernetes.io/edge
               operator: DoesNotExist
             - key: node-role.kubernetes.io/agent
               operator: DoesNotExist
             - key: node-role.kubernetes.io/control-plane
               operator: Exists
     containers:
     - args:
       - --cert-dir=/tmp
       - --secure-port=10250
       - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
       - --kubelet-use-node-status-port
       - --metric-resolution=15s
       - --kubelet-insecure-tls  # 1. 添加以下内容
       - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
       - --requestheader-username-headers=X-Remote-User
       - --requestheader-group-headers=X-Remote-Group
       - --requestheader-extra-headers-prefix=X-Remote-Extra-
       image: registry.cn-hangzhou.aliyuncs.com/59izt-publics/metrics-server:v0.9.0    # 2. 修改镜像使用自己同步的阿里云镜像
       ...
       ports:
       - containerPort: 10250
         name: https
         protocol: TCP
       ...
       volumeMounts:
       - mountPath: /tmp
         name: tmp-dir
       - mountPath: /etc/kubernetes/pki    # 3. 挂载卷到容器
         name: ca-ssl
       nodeSelector:
         kubernetes.io/os: linux
       ...
       volumes:
       - emptyDir: {}
         name: tmp-dir
       - name: ca-ssl   # 4. 挂载证书到卷
         hostPath:
           path: /etc/kubernetes/pki

3. 使用 kubectl 安装 metrics Server

   kubectl create -f components.yaml 

4. 查看 pod 运行状态

   kubectl get pods -n kube-system -l k8s-app=metrics-server
   NAME                              READY   STATUS    RESTARTS   AGE
   metrics-server-69c977b6ff-6ncw4   1/1     Running   0          4m7s

5. 查看集群指标

   # kubectl top node
   NAME          CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
   k8smaster01   192m         13%      1311Mi          46%         
   k8smaster02   182m         13%      1041Mi          36%         
   k8smaster03   153m         10%      1198Mi          42%         
   k8snode01     55m          3%       586Mi           20%         
   k8snode02     67m          4%       659Mi           23%         
   k8snode03     76m          5%       824Mi           28%      

安装 Dashboard

注意：Kubernetes Dashboard 项目已归档，不再进行积极维护。 对于新 Web 工具的选择，请考虑使用 Headlamp

1. 添加 Helm 仓库，并下载 headlamp 包到本地

   helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
   helm pull headlamp/headlamp
   tar -xf headlamp-*.tgz 

2. 修改镜像仓库为国内镜像仓库

   image:
     # -- Container image registry
     registry: registry.cn-hangzhou.aliyuncs.com
     # -- Container image name
     repository: 59izt-publics/headlamp
     # -- Image pull policy. One of Always, Never, IfNotPresent
     pullPolicy: IfNotPresent
     # -- Container image tag, If "" uses appVersion in Chart.yaml
     tag: "v0.46.0"

3. 安装 headlamp

   # helm install -n kube-system headlamp . 
   NAME: headlamp
   LAST DEPLOYED: Mon Oct 19 15:49:28 2026
   NAMESPACE: kube-system
   STATUS: deployed
   REVISION: 1
   DESCRIPTION: Install complete
   TEST SUITE: None
   NOTES:
   1. Get the application URL by running these commands:
     export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=headlamp,app.kubernetes.io/instance=headlamp" -o jsonpath="{.items[0].metadata.name}")
     export CONTAINER_PORT=$(kubectl get pod --namespace kube-system $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
     echo "Visit http://127.0.0.1:8080 to use your application"
     kubectl --namespace kube-system port-forward $POD_NAME 8080:$CONTAINER_PORT
   2. Get the token using
     kubectl create token headlamp --namespace kube-system
   
   NOTE:
     image.tag is explicitly set to v0.46.0.
     Chart upgrades do not change this pinned value automatically.
     If the UI shows an older version than the chart's appVersion, update image.tag explicitly, for example:
     helm upgrade headlamp headlamp/headlamp --namespace kube-system --reuse-values --set image.tag=v0.46.0

4. 获取集群访问令牌

   kubectl create token headlamp --namespace kube-system

5. 将 headlamp 服务端口映射到本地

   kubectl -n kube-system port-forward services/headlamp --address 0.0.0.0 --address ::  8082:80

6. 浏览器访问 http://192.168.0.30:8082，输入验证令牌
   

安装 Gateway API

社区版 Ingress-NGINX 已于 2026 年 3 月正式 EOL，不再发布新版本、不再修 Bug、也不再提供 CVE 补丁。 阿里云 ACK、ACS 也在 2026 年 11 月 30 日后停止维护其 Nginx Ingress Controller 组件。 所以这件事已经不是“要不要迁移”，而是“怎么安全、可控地迁移”。

说明：
Kubernetes 项目推荐使用 Gateway 而不是 Ingress。 Ingress API 已经被冻结。

这意味着：

• Ingress API 是正式发布的，并且遵循正式发布 API 的稳定性保证。 Kubernetes 项目没有计划从 Kubernetes 中移除 Ingress。
• Ingress API 不再进行开发，也不会对其进行进一步的更改或更新。

Gateway API 是一组 API 类别，可提供动态基础设施制备和高级流量路由。Gateway API 通过使用可扩展的、角色导向的、 协议感知的配置机制来提供网络服务。它是一个附加组件， 包含可提供动态基础设施配置和高级流量路由的 API 类别。

Gateway API 具有四种稳定的 API 类别：

• GatewayClass： 定义一组具有配置相同的网关，由实现该类的控制器管理。
• Gateway： 定义流量处理基础设施（例如云负载均衡器）的一个实例。
• HTTPRoute： 定义特定于 HTTP 的规则，用于将流量从 Gateway 监听器映射到后端网络端点的某种呈现。 这些端点通常表示为 Service。
• GRPCRoute： 定义特定于 gRPC 的规则，用于将流量从 Gateway 监听器映射到后端网络端点的某种呈现。 这些端点通常表示为 Service。

Gateway API 被组织成不同的 API 类别，这些 API 类别具有相互依赖的关系，以支持组织中角色导向的特点。 一个 Gateway 对象只能与一个 GatewayClass 相关联；GatewayClass 描述负责管理此类 Gateway 的网关控制器。 各个（可以是多个）路由类别（例如 HTTPRoute）可以关联到此 Gateway 对象。 Gateway 可以对能够挂接到其 listeners 的路由进行过滤，从而与路由形成双向信任模型。

标准发布渠道包括所有已升级到 GA 或 beta 的资源，包括 GatewayClass、Gateway、HTTPRoute 和 ReferenceGrant。
实验性发布渠道包含标准发布渠道中的所有内容，以及一些实验性资源和字段。这包括 TCPRoute、TLSRoute 和 UDPRoute。

请注意，API 的未来版本可能包含对实验性资源和字段的重大更改。例如，任何实验性资源或字段都可能在未来版本中被移除。有关实验性渠道的更多信息，请参阅我们的版本控制文档。

1. 请运行以下 kubectl 命令安装 Gateway API：

   # 安装标准版发布渠道
   kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/standard-install.yaml
   
   # 安装实验性发布渠道
   kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/experimental-install.yaml

2. 查看 Gateway CRDs

   kubectl get crd gateways.gateway.networking.k8s.io