Kube-prometheus 监控 k8s 集群外部的主机

公司的 k8s 集群使用的是 kube-prometheus 监控的，除了 kube-prometheus 外，公司还有一些其他主机也需要使用 Prometheus 监控起来。Prometheus 可以添加额外的配置文件，详细的可以查看 Additional Scrape Configuration

具体的步骤如下:

• 在 additional-scrape-config 里面添加 job;
• 使用 file_sd 去发现主机就可以了;
• 使用 configmap 挂载 file 到容器里面;
• 修改 configmap 就可以添加新的监控主机到 Prometheus;

详细操作步骤

1. 创建 prometheus-additional.yaml 文件，内容如下

   - job_name: 'node-exporter-outer'
     file_sd_configs:
     - files: ['/etc/prometheus/hosts/hosts.yaml']  
       refresh_interval: 5s
     metrics_path: /metrics
     relabel_configs:
     - source_labels: [__address__]
       regex: (.*)
       target_label: instance
       replacement: $1
     - source_labels: [__address__]
       regex: (.*)
       target_label: __address__
       replacement: $1:9100

2. 创建外部主机配置文件 prometheus-external-node-exporter-hosts-cm.yaml，内容如下

   - labels:
       env: "prod"
       hostname: "mysql01"
       app: "mysql"
       role: "master"
     targets:
     - '192.168.10.51'
   - labels:
       env: "prod"
       hostname: "mysql01"
       app: "mysql"
       role: "slave"
     targets:
     - '192.168.10.52'
   - labels:
       env: "prod"
       hostname: "prod-middleware01"
       app: "middleware"
     targets:
     - '192.168.10.38'
   - labels:
       env: "prod"
       hostname: "prod-middleware02"
       app: "middleware"
     targets:
     - '192.168.10.39'
   - labels:
       env: "prod"
       hostname: "prod-middleware03"
       app: "middleware"
     targets:
     - '192.168.10.40'

3. 使用 prometheus-additional.yaml 文件创建 Secret

   kubectl create secret generic additional-configs \
     --from-file=prometheus-additional.yaml \
     -n monitoring

4. 使用 prometheus-external-node-exporter-hosts-cm.yaml 创建 configMap

   kubectl create cm prometheus-external-node-exporter-hosts-cm \
     --from-file=hosts.yaml=prometheus-external-node-exporter-hosts-cm.yaml \
     -n monitoring

5. 修改 prometheus-prometheus.yaml 文件，添加挂载相关配置，如下

   apiVersion: monitoring.coreos.com/v1
   kind: Prometheus
   metadata:
     labels:
       prometheus: k8s
     name: k8s
     namespace: monitoring
   spec:
     alerting:
       alertmanagers:
       - name: alertmanager-main
         namespace: monitoring
         port: web
     image: quay.io/prometheus/prometheus:v2.22.1
     nodeSelector:
       kubernetes.io/os: linux
     podMonitorNamespaceSelector: {}
     podMonitorSelector: {}
     probeNamespaceSelector: {}
     probeSelector: {}
     replicas: 2
     resources:
       requests:
         memory: 400Mi
     ruleSelector:
       matchLabels:
         prometheus: k8s
         role: alert-rules
     securityContext:
       fsGroup: 2000
       runAsNonRoot: true
       runAsUser: 1000
     serviceAccountName: prometheus-k8s
     serviceMonitorNamespaceSelector: {}
     serviceMonitorSelector: {}
     version: v2.22.1
     # 增加以下配置挂载文件到 pod 内部配置
     volumeMounts:
     - mountPath: /etc/prometheus/hosts
       name: prometheus-external-node-exporter-hosts-cm
       readOnly: true
     volumes:
     - configMap:
         name: prometheus-external-node-exporter-hosts-cm
       name: prometheus-external-node-exporter-hosts-cm
     # 增加 additionalScrapeConfigs 相关配置
     additionalScrapeConfigs:
       key: prometheus-additional.yaml
       name: additional-configs
       optional: true

6. 应用资源清单更新

   kubectl replace -f prometheus-prometheus.yaml 

动态更新 Prometheus 配置文件

1. 修改 prometheus-external-node-exporter-hosts-cm.yaml 文件动态更新 Prometheus 配置

   kubectl create cm prometheus-external-node-exporter-hosts-cm \
     --from-file=hosts.yaml=prometheus-external-node-exporter-hosts-cm.yaml \
     --dry-run=client \
     -oyaml | kubectl replace -f - -n monitoring

2. 修改 prometheus-additional.yaml 文件后动态更新 Prometheus 配置

   kubectl create secret generic additional-configs \
     --from-file=prometheus-additional.yaml \
     --dry-run=client \
     -oyaml | kubectl replace -f - -n monitoring