Prothmeues 設定blackbox exporter黑盒監控

Blackbox Exporter是Prometheus社區提供的官方黑盒監控解決方案，其允許用戶通過：HTTP、HTTPS、DNS、TCP以及ICMP的方式對網絡進行探測。
黑盒監控即以用戶的身份測試服務的外部可見性，常見的黑盒監控包括HTTP探針、TCP探針等用於檢測站點或者服務的可訪問性，以及訪問效率等。
黑盒監控相較於白盒監控最大的不同在於黑盒監控是以故障為導向當故障發-生時，黑盒監控能快速發現故障，而白盒監控則側重於主動發現或者預測潛在的問題。
一個完善的監控目標是要能夠從白盒的角度發現潛在問題，能夠在黑盒的角度快速發現已經發生的問題。

# Helm Install blackbox-expoter

helm install blackbox prometheus-community/prometheus-blackbox-exporter

開啟blackbox 9115 port會看到metrics

設定 Prometheus

- job_name: 'blackbox-kubernetes-services'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      # Example relabel to probe only some services that have "example.io/should_be_probed = true" annotation
      #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_probed]
      #    action: keep
      #    regex: true
      # 只有service的annotation中配置了 prometheus.io/http_probe=true 的才進行發現
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_http_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-prometheus-blackbox-exporter:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name


    - job_name: 'blackbox-kubernetes-ingresses'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: ingress
      relabel_configs:
      # Example relabel to probe only some ingresses that have "example.io/should_be_probed = true" annotation
      #  - source_labels: [__meta_kubernetes_ingress_annotation_example_io_should_be_probed]
      #    action: keep
      #    regex: true
      # 只有ingress的annotation中配置了 prometheus.io/http_probe=true的才進行發現
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_http_probe]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        replacement: ${1}://${2}${3}
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-prometheus-blackbox-exporter:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_ingress_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_ingress_name]
        target_label: kubernetes_nam


    - job_name: "blackbox-kubernetes-service-dns"
      metrics_path: /probe # 不是 metrics，是 probe
      params:
        module: [dns] # 使用 DNS 模塊
      static_configs:
      - targets:
        - kube-dns.kube-system:53  # 不要省略端口號 
      relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-prometheus-blackbox-exporter:9115  # 服務地址，和上面的Service定義保持一致

    - job_name: 'blackbox-exporter'
      scrape_timeout: 20s
      metrics_path: /probe
      params:
        module: [http_2xx]  # Look for a HTTP 200 response.
      static_configs:
        - targets:
            - https://kubernetes.io/
            - https://github.com/
      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: blackbox-prometheus-blackbox-exporter:9115  # The blackbox exporter's real hostname:port.

service和ingress需加入以下

apiVersion: v1                                                                                                                                                                                                                            
  kind: Service
  metadata:
    annotations:
      prometheus.io/http-probe: "true"

apiVersion: networking.k8s.io/v1                                                                                                                                                                                                          
  kind: Ingress
  metadata:
    name: download
    annotations:
      prometheus.io/http-probe: "true"

kubectl apply -f prometheus-server-conf.yaml

重啟Prometheus deployment

成功的話會出現在Targets上面

curl測試blackbox是否正常

curl -s "http://blackbox-prometheus-blackbox-exporter:9115/probe?target=www.itnotetk.com&module=http_2xx" |tail -5

# TYPE probe_success gauge
probe_success 1
# HELP probe_tls_version_info Contains the TLS version used
# TYPE probe_tls_version_info gauge
probe_tls_version_info{version="TLS 1.3"} 1

Grafana配置dashboard