1、RabbitMQ介紹

RabbitMQ是實現了高級訊息佇列協定AMQP的開源消息代理軟體(亦稱面向消息的中介軟體)。RabbitMQ伺服器是用Erlang語言編寫的,而集群和容錯移轉是構建在開放電信平臺框架上的。AMQP:Advanced Message Queue,高級訊息佇列協議。它是應用層協議的一個開放標準,為面向消息的中介軟體設計,基於此協定的用戶端與消息中介軟體可傳遞消息,並不受產品、開發語言燈條件的限制
AMQP具有如下的特性:
• 可靠性Reliablity:使用了一些機制來保證可靠性,比如持久化、傳輸確認、發佈確認
• 靈活的路由Flexible Routing:在消息進入佇列之前,通過Exchange來路由消息。對於典型的路由功能,Rabbit已經提供了一些內置的Exchange來實現。針對更複雜的路由功能,可以將多個Exchange綁定在一起,也通過外掛程式機制實現自己的Exchange
• 消息集群Clustering:多個RabbitMQ伺服器可以組成一個集群,形成一個邏輯Broker
• 高可用Highly Avaliable Queues:佇列可以在集群中的機器上進行鏡像,使得在部分節點出問題的情況下佇列仍然可用
• 多種協定Multi-protocol:支援多種訊息佇列協定,如STOMP、MQTT等
• 多種語言用戶端Many Clients:幾乎支援所有常用語言,比如Java、.NET、Ruby等
• 管理介面Management UI:提供了易用的使用者介面,使得使用者可以監控和管理消息Broker的許多方面
• 跟蹤機制Tracing:如果消息異常,RabbitMQ提供了消息的跟蹤機制,使用者可以找出發生了什麼
• 外掛程式機制Plugin System:提供了許多外掛程式,來從多方面進行擴展,也可以編輯自己的外掛程式
2、RabbitMQ的持久化和鏡像佇列
RabbitMQ持久化分為Exchange、Queue、Message
• Exchange和Queue持久化:指持久化Exchange、Queue中繼資料,持久化的是自身,服務宕機Exchange和Queue自身就沒有了
• Message持久化:顧名思義就是把每一條消息體持久化,服務宕機,消息不丟失
RabbitMQ的佇列Queue鏡像,指master node在接受到請求後,會同步到其他節點上,以此來保證高可用。在confirm模式下,具體過程如下
clientpublisher發送消息 –> master node接到消息 –> master node將消息持久化到磁片 –> 將消息非同步發送給其他節點 –> master將ack返回給client publisher
3、RabbitMQ集群在k8s中的部署
將RabbitMQ以集群的方式部署在k8s中,前提是RabbitMQ的每個節點都能像傳統方式一樣進行相互的服務發現。因此RabbitMQ在k8s集群中通過rabbitmq_peer_discovery_k8s plugin與k8s apiserver進行交互,獲取各個服務的URL,且RabbitMQ在k8s集群中必須用statefulset和headless service進行匹配
需要注意的是,rabbitmq_peer_discovery_k8s是RabbitMQ官方基於協力廠商開源專案rabbitmq-autocluster開發,對3.7.X及以上版本提供的Kubernetes下的對等發現外掛程式,可實現rabbitmq集群在k8s中的自動化部署,因此低於3.7.X版本請使用rabbitmq-autocluster
3.1 環境介紹
本文部署的版本是3.8.3
默認部署在default命名空間下,
持久化存儲為storageclass動態儲存裝置,底層為nfs提供
鏡像地址rabbitmq:3.8.3-management
以下yaml參考自官方示例https://github.com/rabbitmq/diy-kubernetes-examples
 
3.2 創建configmap
01-rabbitmq-configmap.yaml
 
kind: ConfigMap 
apiVersion: v1 
metadata: 
  name: rabbitmq-cluster-config 
  namespace: default 
  labels: 
    addonmanager.kubernetes.io/mode: Reconcile 
data: 
    enabled_plugins: | 
      [rabbitmq_management,rabbitmq_peer_discovery_k8s]. 
    rabbitmq.conf: | 
      default_user = admin 
      default_pass = [email protected]# 
      ## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more. 
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s 
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local 
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address? 
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible. 
      ## Set to "hostname" to use pod hostnames. 
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME 
      ## environment variable. 
      cluster_formation.k8s.address_type = hostname 
      ## How often should node cleanup checks run? 
      cluster_formation.node_cleanup.interval = 30 
      ## Set to false if automatic removal of unknown/absent nodes 
      ## is desired. This can be dangerous, see 
      ## * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup 
      ## * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ 
      cluster_formation.node_cleanup.only_log_warning = true 
      cluster_partition_handling = autoheal 
      ## See https://www.rabbitmq.com/ha.html#master-migration-data-locality 
      queue_master_locator=min-masters 
      ## See https://www.rabbitmq.com/access-control.html#loopback-users 
      loopback_users.guest = false 
      cluster_formation.randomized_startup_delay_range.min = 0 
      cluster_formation.randomized_startup_delay_range.max = 2 
      # default is rabbitmq-cluster's namespace 
      # hostname_suffix 
      cluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.default.svc.cluster.local 
      # memory 
      vm_memory_high_watermark.absolute = 1GB 
      # disk 
      disk_free_limit.absolute = 2GB
部分參數說明:
• enabled_plugins:聲明開啟的外掛程式名
• default_pass/default_pass:聲明用戶名和密碼(雖然有部分文章記錄可以通過環境變數的方式聲明,但是經測試,針對此版本如果指定了configmap即rabbitmq的設定檔,聲明的環境變數是沒有用的,都需要在設定檔中指定)
• cluster_formation.k8s.address_type:從k8s返回的Pod容器列表中計算對等節點列表,這裡只能使用主機名稱,官方示例中是ip,但是預設情況下在k8s中pod的ip都是不固定的,因此可能導致節點的配置和資料丟失,後面的yaml中會通過引用中繼資料的方式固定pod的主機名稱。
更多參數請參考官方文檔,這裡就不贅述了
 
3.3 創建service
02-rabbitmq-service.yaml
 
kind: Service 
apiVersion: v1 
metadata: 
  labels: 
    app: rabbitmq-cluster 
  name: rabbitmq-cluster 
  namespace: default 
spec: 
  clusterIP: None 
  ports: 
  - name: rmqport 
    port: 5672 
    targetPort: 5672 
  selector: 
    app: rabbitmq-cluster 

--- 
kind: Service 
apiVersion: v1 
metadata: 
  labels: 
    app: rabbitmq-cluster 
  name: rabbitmq-cluster-manage 
  namespace: default 
spec: 
  ports: 
  - name: http 
    port: 15672 
    protocol: TCP 
    targetPort: 15672 
  selector: 
    app: rabbitmq-cluster 
  type: NodePort
上面定義了兩個Service,一個是rabbitmq的服務埠,一個是管理介面的埠,使用者外部訪問,這裡通過NodePort方式進行暴露
 
3.4 創建rbac授權
前面的介紹中提到了RabbitMQ通過外掛程式與k8s apiserver交互獲得集群中節點相關資訊,因此需要對其進行RBAC授權
 
03-rabbitmq-rbac.yaml
 
apiVersion: v1 
kind: ServiceAccount 
metadata: 
  name: rabbitmq-cluster 
  namespace: default 
--- 
kind: Role 
apiVersion: rbac.authorization.k8s.io/v1beta1 
metadata: 
  name: rabbitmq-cluster 
  namespace: default 
rules: 
- apiGroups: [""] 
  resources: ["endpoints"] 
  verbs: ["get"] 
--- 
kind: RoleBinding 
apiVersion: rbac.authorization.k8s.io/v1beta1 
metadata: 
  name: rabbitmq-cluster 
  namespace: default 
roleRef: 
  apiGroup: rbac.authorization.k8s.io 
  kind: Role 
  name: rabbitmq-cluster 
subjects: 
- kind: ServiceAccount 
  name: rabbitmq-cluster 
  namespace: default
3.5 創建statefulset
RabbitMQ在k8s中作為一個有狀態應用進行部署,因此控制器類型為StatefulSet,yaml中還定義了pvc相關內容
04-rabbitmq-cluster-sts.yaml
 
kind: StatefulSet 
apiVersion: apps/v1 
metadata: 
  labels: 
    app: rabbitmq-cluster 
  name: rabbitmq-cluster 
  namespace: default 
spec: 
  replicas: 3 
  selector: 
    matchLabels: 
      app: rabbitmq-cluster 
  serviceName: rabbitmq-cluster 
  template: 
    metadata: 
      labels: 
        app: rabbitmq-cluster 
    spec: 
      containers: 
      - args: 
        - -c 
        - cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.sh 
          rabbitmq-server 
        command: 
        - sh 
        env: 
        - name: TZ 
          value: 'Asia/Shanghai' 
        - name: RABBITMQ_ERLANG_COOKIE 
          value: 'SWvCP0Hrqv43NG7GybHC95ntCJKoW8UyNFWnBEWG8TY=' 
        - name: K8S_SERVICE_NAME 
          value: rabbitmq-cluster 
        - name: POD_IP 
          valueFrom: 
            fieldRef: 
              fieldPath: status.podIP 
        - name: POD_NAME 
          valueFrom: 
            fieldRef: 
              fieldPath: metadata.name 
        - name: POD_NAMESPACE 
          valueFrom: 
            fieldRef: 
              fieldPath: metadata.namespace 
        - name: RABBITMQ_USE_LONGNAME 
          value: "true" 
        - name: RABBITMQ_NODENAME 
          value: [email protected]$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: RABBITMQ_CONFIG_FILE 
          value: /var/lib/rabbitmq/rabbitmq.conf 
        image: rabbitmq:3.8.3-management 
        imagePullPolicy: IfNotPresent 
        livenessProbe: 
          exec: 
            command: 
            - rabbitmq-diagnostics 
            - status 
          # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations. 
          initialDelaySeconds: 60 
          periodSeconds: 60 
          timeoutSeconds: 15 
        name: rabbitmq 
        ports: 
        - containerPort: 15672 
          name: http 
          protocol: TCP 
        - containerPort: 5672 
          name: amqp 
          protocol: TCP 
        readinessProbe: 
          exec: 
            command: 
            - rabbitmq-diagnostics 
            - status 
          initialDelaySeconds: 20 
          periodSeconds: 60 
          timeoutSeconds: 10 
        volumeMounts: 
        - mountPath: /etc/rabbitmq 
          name: config-volume 
          readOnly: false 
        - mountPath: /var/lib/rabbitmq 
          name: rabbitmq-storage 
          readOnly: false 
        - name: timezone 
          mountPath: /etc/localtime 
          readOnly: true 
      serviceAccountName: rabbitmq-cluster 
      terminationGracePeriodSeconds: 30 
      volumes: 
      - name: config-volume 
        configMap: 
          items: 
          - key: rabbitmq.conf 
            path: rabbitmq.conf 
          - key: enabled_plugins 
            path: enabled_plugins 
          name: rabbitmq-cluster-config 
      - name: timezone 
        hostPath: 
          path: /usr/share/zoneinfo/Asia/Shanghai 
  volumeClaimTemplates: 
  - metadata: 
      name: rabbitmq-storage 
    spec: 
      accessModes: 
      - ReadWriteMany 
      storageClassName: "managed-nfs-storage" 
      resources: 
        requests: 
          storage: 2Gi

3.6 部署檢查
➜ rabbitmq-cluster pwd
/Users/ssgeek/Documents/k8s-manifests/rabbitmq-cluster
➜ rabbitmq-cluster ls
01-rabbitmq-configmap.yaml 02-rabbitmq-service.yaml 03-rabbitmq-rbac.yaml 04-rabbitmq-cluster-sts.yaml
➜ rabbitmq-cluster kubectl apply -f .
configmap/rabbitmq-cluster-config created
service/rabbitmq-cluster created
service/rabbitmq-cluster-manage created
serviceaccount/rabbitmq-cluster created
role.rbac.authorization.k8s.io/rabbitmq-cluster created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster created
statefulset.apps/rabbitmq-cluster created

等待一段時間,查看創建的相關資源
➜ rabbitmq-cluster kubectl get po,sts -l app=rabbitmq-cluster
NAME READY STATUS RESTARTS AGE
pod/rabbitmq-cluster-0 1/1 Running 0 3m22s
pod/rabbitmq-cluster-1 1/1 Running 0 2m18s
pod/rabbitmq-cluster-2 1/1 Running 0 2m23s

NAME READY AGE
statefulset.apps/rabbitmq-cluster 3/3 3m25s

查看日誌,從日誌的最後部分觀察集群建立的狀態

➜ rabbitmq-cluster kubectl logs -f rabbitmq-cluster-0 
'/etc/rabbitmq/rabbitmq.conf' -> '/var/lib/rabbitmq/rabbitmq.conf' 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: list of feature flags found: 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] drop_unroutable_metric 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] empty_basic_get_metric 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] implicit_default_bindings 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] quorum_queue 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] virtual_host_metadata 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: feature flag states written to disk: yes 
2021-02-17 03:30:39.544 [info] <0.269.0> ra: meta data store initialised. 0 record(s) recovered 
2021-02-17 03:30:39.547 [info] <0.274.0> WAL: recovering [] 
2021-02-17 03:31:10.676 [info] <0.313.0> 
 Starting RabbitMQ 3.8.3 on Erlang 22.3.4.1 
 Copyright (c) 2007-2020 Pivotal Software, Inc. 
 Licensed under the MPL 1.1. Website: https://rabbitmq.com 

  ## ## RabbitMQ 3.8.3 
  ## ## 
  ########## Copyright (c) 2007-2020 Pivotal Software, Inc. 
  ###### ## 
  ########## Licensed under the MPL 1.1. Website: https://rabbitmq.com 

  Doc guides: https://rabbitmq.com/documentation.html 
  Support: https://rabbitmq.com/contact.html 
  Tutorials: https://rabbitmq.com/getstarted.html 
  Monitoring: https://rabbitmq.com/monitoring.html 

  Logs: <stdout> 

  Config file(s): /var/lib/rabbitmq/rabbitmq.conf 

  Starting broker...2021-02-17 03:31:10.678 [info] <0.313.0> 
 node : [email protected]ocal 
 home dir : /var/lib/rabbitmq 
 config file(s) : /var/lib/rabbitmq/rabbitmq.conf 
 cookie hash : H+IQL2spD4MDV4jPi7mMAg== 
 log(s) : <stdout> 
 database dir : /var/lib/rabbitmq/mnesia/[email protected]ocal 
2021-02-17 03:31:10.695 [info] <0.313.0> Running boot step pre_boot defined by app rabbit 

...省略中間內容 

2021-02-17 03:31:13.273 [info] <0.824.0> Statistics database started. 
2021-02-17 03:31:13.273 [info] <0.823.0> Starting worker pool 'management_worker_pool' with 3 processes in it 
2021-02-17 03:31:13.467 [info] <0.9.0> Server startup complete; 5 plugins started. 
 * rabbitmq_peer_discovery_k8s 
 * rabbitmq_management 
 * rabbitmq_web_dispatch 
 * rabbitmq_management_agent 
 * rabbitmq_peer_discovery_common 
 completed with 5 plugins. 
2021-02-17 03:32:30.060 [info] <0.566.0> node '[email protected]ocal' up 
2021-02-17 03:32:31.264 [info] <0.566.0> rabbit on node '[email protected]ocal' up 
2021-02-17 03:33:31.280 [info] <0.566.0> node '[email protected]ocal' up 
2021-02-17 03:33:32.627 [info] <0.566.0> rabbit on node '[email protected]ocal' up 

進入到pod中通過用戶端查看集群狀態

➜ rabbitmq-cluster kubectl exec -it rabbitmq-cluster-0 bash 
[email protected]:/# rabbitmqctl cluster_status 
Cluster status of node [email protected]ocal ... 
Basics 

Cluster name: [email protected]ocal 

Disk Nodes 

[email protected]ocal 
[email protected]ocal 
[email protected]ocal 

Running Nodes 

[email protected]ocal 
[email protected]ocal 
[email protected]ocal 

Versions 

[email protected]ocal: RabbitMQ 3.8.3 on Erlang 22.3.4.1 
[email protected]ocal: RabbitMQ 3.8.3 on Erlang 22.3.4.1 
[email protected]ocal: RabbitMQ 3.8.3 on Erlang 22.3.4.1 

Alarms 

(none) 

Network Partitions 

(none) 

Listeners 

Node: [email protected]ocal, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication 
Node: [email protected]ocal, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 
Node: [email protected]ocal, interface: [::], port: 15672, protocol: http, purpose: HTTP API 
Node: [email protected]ocal, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication 
Node: [email protected]ocal, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 
Node: [email protected]ocal, interface: [::], port: 15672, protocol: http, purpose: HTTP API 
Node: [email protected]ocal, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication 
Node: [email protected]ocal, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 
Node: [email protected]ocal, interface: [::], port: 15672, protocol: http, purpose: HTTP API 

Feature flags 

Flag: drop_unroutable_metric, state: enabled 
Flag: empty_basic_get_metric, state: enabled 
Flag: implicit_default_bindings, state: enabled 
Flag: quorum_queue, state: enabled 
Flag: virtual_host_metadata, state: enabled 
通過NodePort訪問管理介面
kubectl get svc -l app=rabbitmq-cluster
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rabbitmq-cluster ClusterIP None 5672/TCP 8m47s
rabbitmq-cluster-manage NodePort 10.1.239.191 15672:30888/TCP 8m47s
 
進k9s看狀態是否為Running
 

確認pvc持久性檔案已建立

mq新增一個管理員指令

登入mq ui 可以看到集群已建立

 

點閱: 74

By tony

自由軟體愛好者~喜歡不斷的思考各種問題,有新的事物都會想去學習嘗試 做實驗並熱衷研究 沒有所謂頂天的技術 只有謙虛及不斷的學習 精進專業,本站主要以分享系統及網路相關知識、資源而建立。 Github http://stnet253.github.io

發佈留言

發佈留言必須填寫的電子郵件地址不會公開。

這個網站採用 Akismet 服務減少垃圾留言。進一步了解 Akismet 如何處理網站訪客的留言資料