Blackbox Exporter是Prometheus社區提供的官方黑盒監控解決方案,其允許用戶通過:HTTP、HTTPS、DNS、TCP以及ICMP的方式對網絡進行探測。
黑盒監控即以用戶的身份測試服務的外部可見性,常見的黑盒監控包括HTTP探針、TCP探針等用於檢測站點或者服務的可訪問性,以及訪問效率等。
黑盒監控相較於白盒監控最大的不同在於黑盒監控是以故障為導向當故障發-生時,黑盒監控能快速發現故障,而白盒監控則側重於主動發現或者預測潛在的問題。
一個完善的監控目標是要能夠從白盒的角度發現潛在問題,能夠在黑盒的角度快速發現已經發生的問題。

# Helm Install blackbox-expoter

helm install blackbox prometheus-community/prometheus-blackbox-exporter

開啟blackbox 9115 port會看到metrics

設定 Prometheus

- job_name: 'blackbox-kubernetes-services'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      # Example relabel to probe only some services that have "example.io/should_be_probed = true" annotation
      #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_probed]
      #    action: keep
      #    regex: true
      # 只有service的annotation中配置了 prometheus.io/http_probe=true 的才進行發現
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_http_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-prometheus-blackbox-exporter:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name


    - job_name: 'blackbox-kubernetes-ingresses'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: ingress
      relabel_configs:
      # Example relabel to probe only some ingresses that have "example.io/should_be_probed = true" annotation
      #  - source_labels: [__meta_kubernetes_ingress_annotation_example_io_should_be_probed]
      #    action: keep
      #    regex: true
      # 只有ingress的annotation中配置了 prometheus.io/http_probe=true的才進行發現
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_http_probe]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        replacement: ${1}://${2}${3}
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-prometheus-blackbox-exporter:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_ingress_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_ingress_name]
        target_label: kubernetes_nam


    - job_name: "blackbox-kubernetes-service-dns"
      metrics_path: /probe # 不是 metrics,是 probe
      params:
        module: [dns] # 使用 DNS 模塊
      static_configs:
      - targets:
        - kube-dns.kube-system:53  # 不要省略端口號 
      relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-prometheus-blackbox-exporter:9115  # 服務地址,和上面的Service定義保持一致

    - job_name: 'blackbox-exporter'
      scrape_timeout: 20s
      metrics_path: /probe
      params:
        module: [http_2xx]  # Look for a HTTP 200 response.
      static_configs:
        - targets:
            - https://kubernetes.io/
            - https://github.com/
      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: blackbox-prometheus-blackbox-exporter:9115  # The blackbox exporter's real hostname:port.

service和ingress需加入以下

apiVersion: v1                                                                                                                                                                                                                            
  kind: Service
  metadata:
    annotations:
      prometheus.io/http-probe: "true"
apiVersion: networking.k8s.io/v1                                                                                                                                                                                                          
  kind: Ingress
  metadata:
    name: download
    annotations:
      prometheus.io/http-probe: "true"

 

kubectl apply -f prometheus-server-conf.yaml

重啟Prometheus deployment

成功的話會出現在Targets上面

 

curl測試blackbox是否正常

curl -s "http://blackbox-prometheus-blackbox-exporter:9115/probe?target=www.itnotetk.com&module=http_2xx" |tail -5

# TYPE probe_success gauge
probe_success 1
# HELP probe_tls_version_info Contains the TLS version used
# TYPE probe_tls_version_info gauge
probe_tls_version_info{version="TLS 1.3"} 1

Grafana配置dashboard

 

Hits: 56

By tony

自由軟體愛好者~喜歡不斷的思考各種問題,有新的事物都會想去學習嘗試 做實驗並熱衷研究 沒有所謂頂天的技術 只有謙虛及不斷的學習 精進專業,本站主要以分享系統及網路相關知識、資源而建立。 Github http://stnet253.github.io

發佈留言

發佈留言必須填寫的電子郵件地址不會公開。 必填欄位標示為 *

這個網站採用 Akismet 服務減少垃圾留言。進一步了解 Akismet 如何處理網站訪客的留言資料