2022-12-08

Prometheus实现邮件+钉钉告警

最近闲着没事，刚好公司买了三台服务器，就随手研究和部署了一下Prometheus（普罗米修斯），研究了下怎么监听这些服务器的进程，当进程挂了怎么告警，记录一下

一、环境和软件

1、CentOS7.9（我这里台服务器都是7.9）
2、Prometheus软件包
3、Node-Exporter软件包
4、Process-Exporter软件包
5、Alertmanager软件包
6、Grafana软件包
7、钉钉webhook插件
8、邮箱
9、钉钉
10、服务器信息：
192.168.1.1 master
192.168.1.2 node1
192.168.1.3 node2

11、软件包官方下载地址：
Prometheus下载地址：https://prometheus.io/download/
Node-Exporter下载地址：https://prometheus.io/download/
Process-Exporter下载地址：https://github.com/ncabatoff/process-exporter/releases
Alertmanager下载地址：https://prometheus.io/download/
Grafana软件包：https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1
钉钉webhook插件下载地址：https://github.com/timonwong/prometheus-webhook-dingtalk/releases/

二、在Master（192.168.1.1）上安装Prometheus(普罗米修斯）

1、解压Prometheus软件包

1
2
3

cd /usr/local/
tar -zxvf prometheus-2.40.5.linux-amd64.tar.gz
mv prometheus-2.40.5.linux-amd64 prometheus

三、分别在node1(192.168.1.2)和node2(192.168.1.3)上安装node-exporter和process-exporter

1、解压node-exporter和process-exporter软件包

cd /usr/local/
tar -zxvf node_exporter-1.5.0.linux-amd64.tar.gz
mv node_exporter-1.5.0.linux-amd64 node_exporter
tar -zxvf process-exporter-0.7.10.linux-amd64.tar.gz
mv process-exporter-0.7.10.linux-amd64 process-exporter

四、分别在node1和node2上配置process-exporter

默认process-exporter解压后里面是没有任何配置文件的，只有一个LICENSE一个README.md一个process-exporter可执行文件，我们需要手动创建一个.yaml的配置文件

1	vim process-exporter.yaml //这里的yaml文件么可以自己随意取

然后把这东西贴进去

process_names:
  - name: "{{.Matches}}"
    cmdline:
    - 'nginx'

  - name: "{{.Matches}}"
    cmdline:
    - 'mysqld'

参数说明：

 {{.Comm}} 包含原始可执行文件的basename，/proc//stat 中的换句话说，2nd 字段

{{.ExeBase}} 包含可执行文件的basename

{{.ExeFull}} 包含可执行文件的完全限定路径

{{.Username}} 包含有效用户的用户名

{{.Matches}} 映射包含应用命令行tlb所产生的所有匹配项

 process_names中的每个项目都必须包含一个或多个选择器(comm，exe或cmdline), 如果存在多个选择器，则它们必须全部匹配。 每个选择器都是一个字符串列表，用于与进程的comm，argv [0]匹配；对于cmdline，则是一个适用于命令行的正则表达式。 cmdline regexp使用Go语法。

对于comm和exe，字符串列表是一个OR，这意味着与任何字符串匹配的任何进程都将添加到该项目的组中。

对于cmdline，正则表达式列表为AND，表示它们都必须匹配。 正则表达式中的任何捕获组都必须使用？P <name>选项为捕获分配一个名称，该名称用于填充.Matches。

例如：> ps -ef | grep redis redis 771 1 0 Jun05 ? 00:45:49 /usr/bin/redis-server *:6379

戴戴的Linux

五、在Master（192.168.1.1）上分别安装Alertmanager和钉钉webhook插件

1、解压Alertmanager和钉钉webhook插件

cd /usr/local
tar -zxvf alertmanager-0.24.0.linux-amd64.tar.gz.
mv alertmanager-0.24.0.linux-amd64.tar.gz alertmanager
tar -zxvf prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz
mv prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz prometheus-webhook-dingtalk

六、获取钉钉的webhook链接

戴戴的Linux

七、获取邮箱发件信息

1、开启smtp功能，我这里用的是163

戴戴的Linux

获取到授权码后，要把授权码记住，等会用得到

八、修改钉钉告警插件配置文件

1、修改配置文件

1 2	cd /usr/local/prometheus-webhook-dingtalk/ vim config.example.yml

## Request timeout
# timeout: 5s

## Uncomment following line in order to write template from scratch (be careful!)
#no_builtin_template: true

## Customizable templates path
templates:
  - /usr/local/prometheus-webhook-dingtalk/contrib/templates/template.tmpl  //这个是钉钉的告警模板位置


## You can also override default template using `default_message`
## The following example to use the 'legacy' template from v0.3.0
#default_message:
#  title: '{{ template "legacy.title" . }}'
#  text: '{{ template "legacy.content" . }}'

## Targets, previously was known as "profiles"
targets:
  webhook1:
    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  //这里
    填刚才钉钉拿到的那个webhook地址  
    # secret for signature
    secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  //这里填刚才加签的密钥
  #ebhook2:
  # url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
  #ebhook_legacy:
  #  url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
    # Customize template content
    message:
      # Use legacy template
      title: '{{ template "ops.title" . }}'
      text: '{{ template "ops.content" . }}'
  #ebhook_mention_all:
  # url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
  # mention:
  #   all: true
  #ebhook_mention_users:
  # url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
  # mention:
  #   mobiles: ['156xxxx8827', '189xxxx8325']

2、修改告警模板

1
2
3

/usr/local/prometheus-webhook-dingtalk/contrib/templates/legacy
mv template.tmpl template.bak
vim template.tmpl

把下面的贴进去

{{ define "__subject" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{ end }}


{{ define "__alert_list" }}{{ range . }}
---
{{ if .Labels.owner }}@{{ .Labels.owner }}{{ end }}

**告警名称**: {{ index .Annotations "title" }}

**告警级别**: {{ .Labels.severity }}

**告警主机**: {{ .Labels.instance }}

**告警信息**: {{ index .Annotations "description" }}

**告警时间**: {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai" }}
{{ end }}{{ end }}

{{ define "__resolved_list" }}{{ range . }}
---
{{ if .Labels.owner }}@{{ .Labels.owner }}{{ end }}

**告警名称**: {{ index .Annotations "title" }}

**告警级别**: {{ .Labels.severity }}

**告警主机**: {{ .Labels.instance }}

**告警信息**: {{ index .Annotations "description" }}

**告警时间**: {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai" }}

**恢复时间**: {{ dateInZone "2006.01.02 15:04:05" (.EndsAt) "Asia/Shanghai" }}
{{ end }}{{ end }}


{{ define "default.title" }}
{{ template "__subject" . }}
{{ end }}

{{ define "default.content" }}
{{ if gt (len .Alerts.Firing) 0 }}
**====侦测到{{ .Alerts.Firing | len  }}个故障====**
{{ template "__alert_list" .Alerts.Firing }}
---
{{ end }}

{{ if gt (len .Alerts.Resolved) 0 }}
**====恢复{{ .Alerts.Resolved | len  }}个故障====**
{{ template "__resolved_list" .Alerts.Resolved }}
{{ end }}
{{ end }}


{{ define "ding.link.title" }}{{ template "default.title" . }}{{ end }}
{{ define "ding.link.content" }}{{ template "default.content" . }}{{ end }}
{{ template "default.title" . }}
{{ template "default.content" . }}

九、配置Alertmanager

1、我们需要创建一个告警规则，默认没有，需要手动创建一个

1 2	cd /usr/local/alertmanager/ vim alertinfo.yml //这里alertinfo名字是我自己取的，你也可以命名成你喜欢的名字，但是后缀不能变

把这贴进去

groups:
 - name: nginx
   rules:
   - alert: hbbr
     expr: (namedprocess_namegroup_num_procs{groupname="map[:nginx]"}) == 0
     for: 1m
     labels:
        severity: critical
        status: 非常严重
     annotations:
         summary: "{{ $labels.hostname }} nginx server has been down for more than 1 minutes"
         description: "nginx kill"

 - name: mysqld
   rules:
   - alert: mysqld
     expr: (namedprocess_namegroup_num_procs{groupname="map[:mysqld]"}) == 0
     for: 1m
     labels:
        severity: critical
        status: 非常严重
     annotations:
         summary: "{{ $labels.hostname }} mysqld server has been down for more than 1 minutes"
         description: "mysql kill"

2、修改Alertmanager.yml配置文件

1
2
3

cd /usr/local/alertmanager/
mv alertmanager.yml alertmanager.yml.bak
vim alertmanager.yml

这个贴进去

global:
  resolve_timeout: 10s
  smtp_smarthost: 'smtp.163.com:465'
  smtp_from: 'xxxxxx@163.com'  //这里填写刚才那个163的邮箱号
  mtp_auth_username: 'xxxxxx@163.com'  //这里填写刚才那个163的邮箱号
  smtp_auth_password: 'AIHWJKFLAJWAKBNF'  //这里填写刚才那个163的邮箱获取到的授权码
  smtp_require_tls: false
templates:
  - /usr/local/alertmanager/templatest/*tmpl
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 1m
  receiver: 'mail'
receivers:
- name: 'mail'
  email_configs:
  - to: 'xxxxxx@qq.com'这里填写接收邮件告警的邮箱
    send_resolved: true
    headers: { Subject: "[WARNING]系统告警邮件" }
  webhook_configs:
  - url: 'http://192.168.1.1:8060/dingtalk/webhook1/send'
    send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

十、配置Prometheus（普罗米修斯）

1
2
3

cd /usr/local/prometheus/
mv prometheus.yml prometheus.yml.bak
vim prometheus.yml

这个贴进去

global:
  scrape_interval: 15s 1 minute.
  evaluation_interval: 15s 

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["192.168.1.1:9093"]
'evaluation_interval'.
rule_files:
  - "/usr/local/alertmanager/alertinfo.yml"
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # scheme defaults to 'http'.

    static_configs:
     - targets: ["192.168.1.1:9090"]

  - job_name: 'node1监听'
    static_configs:
    - targets: ['192.168.1.2:2235']
      labels:
        hostname: node1监听
        host_ip: 192.168.1.2
        
  - job_name: 'node2监听'
    static_configs:
    - targets: ['192.168.1.2:2235']
      labels:
        hostname: node2监听
        host_ip: 192.168.1.2

  - job_name: 'node1'
    static_configs:
    - targets: ['192.168.1.2:1235']

  - job_name: 'node2'
    static_configs:
    - targets: ['192.168.1.3:1235']

十一、登陆后台查看节点情况

Master：http://192.168.1.1:9090

戴戴的Linux

全部都是up状态，说明有监听到，而且服务正常

十二、模拟服务中断告警

随便进到一个节点把刚才监听的nginx或者mysqld杀了，这时我们会收到钉钉和邮件的告警

1	systemctl stop nginx

戴戴的Linux

prometheus和alertmanager地址

prometheus：http://192.168.1.1:9090
alertmanager：http://192.168.1.1:9093

十三、部署Grafana

1、解压grafana

1
2
3

cd /usr/local/
tar -zxvf grafana-enterprise-9.3.1.linux-amd64.tar.gz
mv grafana-enterprise-9.3.1.linux-amd64 grafana

2、启动grafana

1	./grafana-server -homepath /usr/local/grafana

访问地址：192.168.1.1:3000

十四、在master给这些服务写个服务并设置开机自启

1、给prometheus写个服务并设置开机自启

1	vim /usr/lib/systemd/system/prometheus.service

这个贴进去

[Unit]
Description=Prometheus Monitoring System
Documentation=Prometheus Monitoring System

[Service]
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --web.listen-address=:9090
Restart=on-failure
[Install]
WantedBy=multi-user.target

设置开机自启并启动、查看状态

1 2	systemctl daemon-reload systemctl enable prometheus && systemctl restart prometheus && systemctl status prometheus

2、给alertmanager写个服务并设置开机自启

1	vim /usr/lib/systemd/system/alertmanager.service

这个贴进去

[Unit]
Description=alertmanager
Documentation=alertmanager

[Service]
ExecStart=/usr/local/alertmanager/alertmanager --config.file=/usr/local/alertmanager/alertmanager.yml
Restart=on-failure
[Install]
WantedBy=multi-user.target

设置开机自启并启动、查看状态

1 2	systemctl daemon-reload systemctl enable alertmanager && systemctl restart alertmanager && systemctl status alertmanager

3、给grafana写个服务并设置开机自启

1	vim /usr/lib/systemd/system/grafana.service

这个贴进去

[Unit]
Description=grafana
Documentation=grafana

[Service]
ExecStart=/usr/local/grafana/bin/grafana-server -homepath /usr/local/grafana
Restart=on-failure
[Install]
WantedBy=multi-user.target

设置开机自启并启动、查看状态

1 2	systemctl daemon-reload systemctl enable grafana && systemctl restart grafana && systemctl status grafana

4、给dingtalkwebhook写个服务并设置开机自启

1	vim /usr/lib/systemd/system/prometheusdingtalk.service

这个贴进去

[Unit]
Description=prometheus-webhook-dingtalk
Documentation=prometheus-webhook-dingtalk

[Service]
ExecStart=/usr/local/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk  --config.file=/usr/local/prometheus-webhook-dingtalk/config.example.yml
Restart=on-failure
[Install]
WantedBy=multi-user.target

设置开机自启并启动、查看状态

1 2	systemctl daemon-reload systemctl enable prometheusdingtalk && systemctl restart prometheusdingtalk && systemctl status prometheusdingtalk

十五、分别在node1和node2给这些服务写个服务并设置开机自启

1、给node-exporter写个服务并设置开机自启

1	vim /usr/lib/systemd/system/nodeexporter.service

这个贴进去

[Unit]
Description=node-exporter
Documentation=node-exporter

[Service]
ExecStart=/usr/local/node_exporter/node_exporter --web.listen-address=:1235
Restart=on-failure
[Install]
WantedBy=multi-user.target

设置开机自启并启动、查看状态

1 2	systemctl daemon-reload systemctl enable nodeexporter && systemctl restart nodeexporter && systemctl status nodeexporter

2、给process-exporter写个服务并设置开机自启

1	vim /usr/lib/systemd/system/processexporter.service

这个贴进去

[Unit]
Description=Prometheus exporter for processors metrics, written in Go with pluggable metric collectors.
Documentation=https://daishenghui.club
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/usr/local/process-exporter
ExecStart=/usr/local/process-exporter/process-exporter -config.path=/usr/local/process-exporter/process-exporter.yaml --web.listen-address=:2235
Restart=on-failure

[Install]
WantedBy=multi-user.target