Linux 安装部署 Prometheus服务

2023-10-18

字数统计: 1.5k字 | 阅读时长≈ 8分

参考文章: https://www.jianshu.com/p/e59cfd15612e

安装配置

下载安装包

# 下载 Prometheus 服务端程序
wget https://github.com/prometheus/prometheus/releases/download/v2.18.1/prometheus-2.18.1.linux-amd64.tar.gz

# 下载 Prometheus 告警管理程序
wget https://github.com/prometheus/alertmanager/releases/download/v0.20.0/alertmanager-0.20.0.linux-amd64.tar.gz

# 下载节点服务器监控程序
wget https://github.com/prometheus/node_exporter/releases/download/v1.0.0-rc.1/node_exporter-1.0.0-rc.1.linux-amd64.tar.gz

# 下载 Grafana 面板程序
wget https://dl.grafana.com/oss/release/grafana-6.7.3.linux-amd64.tar.gz

安装 Prometheus

解压压缩包

1 2	tar zxvf prometheus-2.18.1.linux-amd64.tar.gz mv prometheus-2.18.1.linux-amd64 /usr/local/prometheus-2.18.1

配置服务文件 /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus-2.18.1/prometheus \
--config.file=/usr/local/prometheus-2.18.1/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.external-url=http://172.16.68.23:9090

[Install]
WantedBy=multi-user.target

加载服务并启动 Prometheus 服务

1
2
3

systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus

查看端口

[root@prometheus local]# netstat -anptl |grep 9090
tcp        0      0 127.0.0.1:60298         127.0.0.1:9090          ESTABLISHED 15894/prometheus    
tcp6       0      0 :::9090                 :::*                    LISTEN      15894/prometheus    
tcp6       0      0 ::1:9090                ::1:35212               ESTABLISHED 15894/prometheus    
tcp6       0      0 ::1:35212               ::1:9090                ESTABLISHED 15894/prometheus    
tcp6       0      0 127.0.0.1:9090          127.0.0.1:60298         ESTABLISHED 15894/prometheus

浏览器访问地址 172.16.68.23:9090

安装 Alertmanager

解压压缩包

1 2	tar zxvf alertmanager-0.20.0.linux-amd64.tar.gz mv alertmanager-0.20.0.linux-amd64 /usr/local/alertmanager-0.20.0

配置服务文件 /etc/systemd/system/alertmanager.service

[Unit]
Description=Alertmanager
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/alertmanager-0.20.0/alertmanager \
--config.file=/usr/local/alertmanager-0.20.0/alertmanager.yml

[Install]
WantedBy=multi-user.target

加载服务并启动 alertmanager 服务

1
2
3

systemctl daemon-reload
systemctl start alertmanager
systemctl enable alertmanager

查看端口

1 2	netstat -anplt \|grep 9093 tcp6 0 0 :::9093 :::* LISTEN 15957/alertmanager

安装 Grafana

解压文件

1	tar zxvf grafana-6.7.3.linux-amd64.tar.gz -C /usr/local

配置服务文件 /etc/systemd/system/grafana.service

[Unit]
Description=Grafana Server
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/grafana-6.7.3/bin/grafana-server \
-config /usr/local/grafana-6.7.3/conf/defaults.ini \
-homepath /usr/local/grafana-6.7.3

[Install]
WantedBy=multi-user.target

重新加载服务并启动 grafana 服务

1
2
3

systemctl daemon-reload
systemctl start grafana
systemctl enable grafana

查看端口

1 2	netstat -anptl \|grep 3000 tcp6 0 0 :::3000 :::* LISTEN 10540/grafana-serve

grafana 初始账号密码为 admin/admin, 第一次登陆会提示修改密码。

安装 exporter

解压文件

1 2	tar zxvf node_exporter-0.18.1.linux-amd64.tar.gz mv node_exporter-0.18.1.linux-amd64 /usr/local/node_exporter-0.18.1

配置服务文件 /etc/systemd/system/node_exporter.service

[Unit]
Description=node_exporter
After=network.target

[Service]
Restart=on-failure
ExecStart=/usr/local/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

重新加载服务并启动 node_exporter 服务

1
2
3

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter

查看端口

1 2	netstat -anptl \|grep 9100 tcp6 0 0 :::9100 :::* LISTEN 16132/node_exporter

配置 Prometheus 监控 node 节点

编辑 Prometheus 配置文件 prometheus.yml，添加监控对象以及监控任务

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'Host'
    file_sd_configs:
    - files:
      - 'configs/host.yml'
    metrics_path: /metrics
    refresh_interval: 15s
    relabel_configs:
    - source_labels: [__address__]
      regex: (.*)
      target_label: instance
      replacement: $1
    - source_labels: [__address__]
      regex: (.*)
      target_label: __address__
      replacement: $1:9100

添加 configs/host.yml 文件

- labels:
  - env_name: "deveploy"
  - soft_name: "test"
  - server_name: "59izt.com"
  targets:
  - '172.16.68.23'
  - '172.16.68.30'

重启服务

1	systemctl restart prometheus

配置 Prometheus Alertmanager

修改 prometheus.yml 配置文件的如下内容

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

rule_files:
  - "rules/host_status.yml"
  - "rules/memory_usage.yml"

创建 rules 文件

1	mkdir rules && cd rules

host_status.yml 内容如下

groups:
- name: 'Host Status Monitor'
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
        severity: 'critical'
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."

memory_usage.yml

groups:
- name: Memory Monitor
  rules:
  - alert: NodeMemoryUsage
    expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes )) / node_memory_MemTotal_bytes * 100 > 80
    for: 1m
    labels:
      severity: 'warning'
    annotations:
      summary: "{{$labels.instance}}: High Memory usage detected"
      description: "{{$labels.instance}}: Memory usage is above 80% (current value is:{{ $value }})"

修改 Alertmanager 配置文件 alertmanager.yml

global:
  # 每2分钟检查一次是否恢复
  resolve_timeout: 2m
  # SMTP的相关配置
  smtp_smarthost: smtp.163.com:25
  smtp_from: xxxxx@163.com
  smtp_auth_username: xxxxx@163.com 
  smtp_auth_password: xxxxxx

# 自定义 通知的模板的 目录 或者 文件.
templates:
  - 'template/*.tmpl'

# 路由树的根节点, 每个传进来的报警从这里开始.
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 30m
  receiver: 'email'

receivers:
- name: 'email'
  email_configs:
  - to: '2350686113@qq.com'
    headers: { Subject: "[WARN] Prometheus 告警邮件" }
    html: '{{ template "email.html" . }}'
    send_resolved: true

  wechat_configs:
  - corp_id: 'xxxxxxx'
    api_secret: 'xxxxxxxxxxxxxxxxxxxx'
    agent_id: 'x'
    to_party: '2'
    message: '{{ template "wechat.html" . }}'
    send_resolved: true

创建告警模板文件

email.tmpl

{{ define "email.html" }}
<table border="1">
  <tr>
    <td>报警项</td>
    <td>实例</td>
    <td>报警阀值</td>
    <td>开始时间</td>
  </tr>
  {{ range $i, $alert := .Alerts }}
  <tr>
    <td>{{ index $alert.Labels "alertname" }}</td>
    <td>{{ index $alert.Labels "instance" }}</td>
    <td>{{ index $alert.Annotations "value" }}</td>
    <td>{{ $alert.StartsAt }}</td>
  </tr>
  {{ end }}
</table>
{{ end }}

wechat.tmpl

{{ define "wechat.html" }}
    {{ range $i, $alert := .Alerts }}
    ===alertmanager监控报警===
    告警状态：{{ .Status }}
    告警级别：{{ $alert.Labels.severity }}
    告警类型：{{ $alert.Labels.alertname }}
    告警应用：{{ $alert.Annotations.summary }}
    故障主机: {{ $alert.Labels.instance }}
    告警主题: {{ $alert.Annotations.summary }}
    触发阀值：{{ $alert.Annotations.value }}
    告警详情: {{ $alert.Annotations.description }}
    触发时间: {{ $alert.StartsAt.Format "2006-01-02 15:04:05" }}
    ===========end============
    {{ end }}
{{ end }}

重启服务

systemctl restart prometheus
systemctl status prometheus
systemctl restart alertmanager
systemctl status alertmanager

测试告警

停掉一台服务器的 node_exporter，然后访问 http://172.16.68.23:9090/alerts, 查看告警相关的规则