Kubernetes 佈署 RabbitMQ

1、RabbitMQ介紹

RabbitMQ是實現了高級訊息佇列協定AMQP的開源消息代理軟體（亦稱面向消息的中介軟體）。RabbitMQ伺服器是用Erlang語言編寫的，而集群和容錯移轉是構建在開放電信平臺框架上的。AMQP：Advanced Message Queue，高級訊息佇列協議。它是應用層協議的一個開放標準，為面向消息的中介軟體設計，基於此協定的用戶端與消息中介軟體可傳遞消息，並不受產品、開發語言燈條件的限制

AMQP具有如下的特性：

• 可靠性Reliablity：使用了一些機制來保證可靠性，比如持久化、傳輸確認、發佈確認

• 靈活的路由Flexible Routing：在消息進入佇列之前，通過Exchange來路由消息。對於典型的路由功能，Rabbit已經提供了一些內置的Exchange來實現。針對更複雜的路由功能，可以將多個Exchange綁定在一起，也通過外掛程式機制實現自己的Exchange

• 消息集群Clustering：多個RabbitMQ伺服器可以組成一個集群，形成一個邏輯Broker

• 高可用Highly Avaliable Queues：佇列可以在集群中的機器上進行鏡像，使得在部分節點出問題的情況下佇列仍然可用

• 多種協定Multi-protocol：支援多種訊息佇列協定，如STOMP、MQTT等

• 多種語言用戶端Many Clients：幾乎支援所有常用語言，比如Java、.NET、Ruby等

• 管理介面Management UI：提供了易用的使用者介面，使得使用者可以監控和管理消息Broker的許多方面

• 跟蹤機制Tracing：如果消息異常，RabbitMQ提供了消息的跟蹤機制，使用者可以找出發生了什麼

• 外掛程式機制Plugin System：提供了許多外掛程式，來從多方面進行擴展，也可以編輯自己的外掛程式

2、RabbitMQ的持久化和鏡像佇列

RabbitMQ持久化分為Exchange、Queue、Message

• Exchange和Queue持久化：指持久化Exchange、Queue中繼資料，持久化的是自身，服務宕機Exchange和Queue自身就沒有了

• Message持久化：顧名思義就是把每一條消息體持久化，服務宕機，消息不丟失

RabbitMQ的佇列Queue鏡像，指master node在接受到請求後，會同步到其他節點上，以此來保證高可用。在confirm模式下，具體過程如下

clientpublisher發送消息 –> master node接到消息 –> master node將消息持久化到磁片 –> 將消息非同步發送給其他節點 –> master將ack返回給client publisher

3、RabbitMQ集群在k8s中的部署

將RabbitMQ以集群的方式部署在k8s中，前提是RabbitMQ的每個節點都能像傳統方式一樣進行相互的服務發現。因此RabbitMQ在k8s集群中通過rabbitmq_peer_discovery_k8s plugin與k8s apiserver進行交互，獲取各個服務的URL，且RabbitMQ在k8s集群中必須用statefulset和headless service進行匹配

需要注意的是，rabbitmq_peer_discovery_k8s是RabbitMQ官方基於協力廠商開源專案rabbitmq-autocluster開發，對3.7.X及以上版本提供的Kubernetes下的對等發現外掛程式，可實現rabbitmq集群在k8s中的自動化部署，因此低於3.7.X版本請使用rabbitmq-autocluster

3.1 環境介紹

本文部署的版本是3.8.3

默認部署在default命名空間下，

持久化存儲為storageclass動態儲存裝置，底層為nfs提供

鏡像地址rabbitmq:3.8.3-management

以下yaml參考自官方示例https://github.com/rabbitmq/diy-kubernetes-examples

3.2 創建configmap

01-rabbitmq-configmap.yaml

kind: ConfigMap 
apiVersion: v1 
metadata: 
  name: rabbitmq-cluster-config 
  namespace: default 
  labels: 
    addonmanager.kubernetes.io/mode: Reconcile 
data: 
    enabled_plugins: | 
      [rabbitmq_management,rabbitmq_peer_discovery_k8s]. 
    rabbitmq.conf: | 
      default_user = admin 
      default_pass = 123!@# 
      ## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more. 
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s 
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local 
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address? 
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible. 
      ## Set to "hostname" to use pod hostnames. 
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME 
      ## environment variable. 
      cluster_formation.k8s.address_type = hostname 
      ## How often should node cleanup checks run? 
      cluster_formation.node_cleanup.interval = 30 
      ## Set to false if automatic removal of unknown/absent nodes 
      ## is desired. This can be dangerous, see 
      ## * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup 
      ## * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ 
      cluster_formation.node_cleanup.only_log_warning = true 
      cluster_partition_handling = autoheal 
      ## See https://www.rabbitmq.com/ha.html#master-migration-data-locality 
      queue_master_locator=min-masters 
      ## See https://www.rabbitmq.com/access-control.html#loopback-users 
      loopback_users.guest = false 
      cluster_formation.randomized_startup_delay_range.min = 0 
      cluster_formation.randomized_startup_delay_range.max = 2 
      # default is rabbitmq-cluster's namespace 
      # hostname_suffix 
      cluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.default.svc.cluster.local 
      # memory 
      vm_memory_high_watermark.absolute = 1GB 
      # disk 
      disk_free_limit.absolute = 2GB

部分參數說明：

• enabled_plugins：聲明開啟的外掛程式名

• default_pass/default_pass：聲明用戶名和密碼（雖然有部分文章記錄可以通過環境變數的方式聲明，但是經測試，針對此版本如果指定了configmap即rabbitmq的設定檔，聲明的環境變數是沒有用的，都需要在設定檔中指定）

• cluster_formation.k8s.address_type：從k8s返回的Pod容器列表中計算對等節點列表，這裡只能使用主機名稱，官方示例中是ip，但是預設情況下在k8s中pod的ip都是不固定的，因此可能導致節點的配置和資料丟失，後面的yaml中會通過引用中繼資料的方式固定pod的主機名稱。

更多參數請參考官方文檔，這裡就不贅述了

3.3 創建service

02-rabbitmq-service.yaml

kind: Service 
apiVersion: v1 
metadata: 
  labels: 
    app: rabbitmq-cluster 
  name: rabbitmq-cluster 
  namespace: default 
spec: 
  clusterIP: None 
  ports: 
  - name: rmqport 
    port: 5672 
    targetPort: 5672 
  selector: 
    app: rabbitmq-cluster 

--- 
kind: Service 
apiVersion: v1 
metadata: 
  labels: 
    app: rabbitmq-cluster 
  name: rabbitmq-cluster-manage 
  namespace: default 
spec: 
  ports: 
  - name: http 
    port: 15672 
    protocol: TCP 
    targetPort: 15672 
  selector: 
    app: rabbitmq-cluster 
  type: NodePort

上面定義了兩個Service，一個是rabbitmq的服務埠，一個是管理介面的埠，使用者外部訪問，這裡通過NodePort方式進行暴露

3.4 創建rbac授權

前面的介紹中提到了RabbitMQ通過外掛程式與k8s apiserver交互獲得集群中節點相關資訊，因此需要對其進行RBAC授權

03-rabbitmq-rbac.yaml

apiVersion: v1 
kind: ServiceAccount 
metadata: 
  name: rabbitmq-cluster 
  namespace: default 
--- 
kind: Role 
apiVersion: rbac.authorization.k8s.io/v1beta1 
metadata: 
  name: rabbitmq-cluster 
  namespace: default 
rules: 
- apiGroups: [""] 
  resources: ["endpoints"] 
  verbs: ["get"] 
--- 
kind: RoleBinding 
apiVersion: rbac.authorization.k8s.io/v1beta1 
metadata: 
  name: rabbitmq-cluster 
  namespace: default 
roleRef: 
  apiGroup: rbac.authorization.k8s.io 
  kind: Role 
  name: rabbitmq-cluster 
subjects: 
- kind: ServiceAccount 
  name: rabbitmq-cluster 
  namespace: default

3.5 創建statefulset

RabbitMQ在k8s中作為一個有狀態應用進行部署，因此控制器類型為StatefulSet，yaml中還定義了pvc相關內容

04-rabbitmq-cluster-sts.yaml

kind: StatefulSet 
apiVersion: apps/v1 
metadata: 
  labels: 
    app: rabbitmq-cluster 
  name: rabbitmq-cluster 
  namespace: default 
spec: 
  replicas: 3 
  selector: 
    matchLabels: 
      app: rabbitmq-cluster 
  serviceName: rabbitmq-cluster 
  template: 
    metadata: 
      labels: 
        app: rabbitmq-cluster 
    spec: 
      containers: 
      - args: 
        - -c 
        - cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.sh 
          rabbitmq-server 
        command: 
        - sh 
        env: 
        - name: TZ 
          value: 'Asia/Shanghai' 
        - name: RABBITMQ_ERLANG_COOKIE 
          value: 'SWvCP0Hrqv43NG7GybHC95ntCJKoW8UyNFWnBEWG8TY=' 
        - name: K8S_SERVICE_NAME 
          value: rabbitmq-cluster 
        - name: POD_IP 
          valueFrom: 
            fieldRef: 
              fieldPath: status.podIP 
        - name: POD_NAME 
          valueFrom: 
            fieldRef: 
              fieldPath: metadata.name 
        - name: POD_NAMESPACE 
          valueFrom: 
            fieldRef: 
              fieldPath: metadata.namespace 
        - name: RABBITMQ_USE_LONGNAME 
          value: "true" 
        - name: RABBITMQ_NODENAME 
          value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: RABBITMQ_CONFIG_FILE 
          value: /var/lib/rabbitmq/rabbitmq.conf 
        image: rabbitmq:3.8.3-management 
        imagePullPolicy: IfNotPresent 
        livenessProbe: 
          exec: 
            command: 
            - rabbitmq-diagnostics 
            - status 
          # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations. 
          initialDelaySeconds: 60 
          periodSeconds: 60 
          timeoutSeconds: 15 
        name: rabbitmq 
        ports: 
        - containerPort: 15672 
          name: http 
          protocol: TCP 
        - containerPort: 5672 
          name: amqp 
          protocol: TCP 
        readinessProbe: 
          exec: 
            command: 
            - rabbitmq-diagnostics 
            - status 
          initialDelaySeconds: 20 
          periodSeconds: 60 
          timeoutSeconds: 10 
        volumeMounts: 
        - mountPath: /etc/rabbitmq 
          name: config-volume 
          readOnly: false 
        - mountPath: /var/lib/rabbitmq 
          name: rabbitmq-storage 
          readOnly: false 
        - name: timezone 
          mountPath: /etc/localtime 
          readOnly: true 
      serviceAccountName: rabbitmq-cluster 
      terminationGracePeriodSeconds: 30 
      volumes: 
      - name: config-volume 
        configMap: 
          items: 
          - key: rabbitmq.conf 
            path: rabbitmq.conf 
          - key: enabled_plugins 
            path: enabled_plugins 
          name: rabbitmq-cluster-config 
      - name: timezone 
        hostPath: 
          path: /usr/share/zoneinfo/Asia/Shanghai 
  volumeClaimTemplates: 
  - metadata: 
      name: rabbitmq-storage 
    spec: 
      accessModes: 
      - ReadWriteMany 
      storageClassName: "managed-nfs-storage" 
      resources: 
        requests: 
          storage: 2Gi

3.6 部署檢查
➜ rabbitmq-cluster pwd
/Users/ssgeek/Documents/k8s-manifests/rabbitmq-cluster
➜ rabbitmq-cluster ls
01-rabbitmq-configmap.yaml 02-rabbitmq-service.yaml 03-rabbitmq-rbac.yaml 04-rabbitmq-cluster-sts.yaml
➜ rabbitmq-cluster kubectl apply -f .
configmap/rabbitmq-cluster-config created
service/rabbitmq-cluster created
service/rabbitmq-cluster-manage created
serviceaccount/rabbitmq-cluster created
role.rbac.authorization.k8s.io/rabbitmq-cluster created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster created
statefulset.apps/rabbitmq-cluster created

等待一段時間，查看創建的相關資源
➜ rabbitmq-cluster kubectl get po,sts -l app=rabbitmq-cluster
NAME READY STATUS RESTARTS AGE
pod/rabbitmq-cluster-0 1/1 Running 0 3m22s
pod/rabbitmq-cluster-1 1/1 Running 0 2m18s
pod/rabbitmq-cluster-2 1/1 Running 0 2m23s

NAME READY AGE
statefulset.apps/rabbitmq-cluster 3/3 3m25s

查看日誌，從日誌的最後部分觀察集群建立的狀態

➜ rabbitmq-cluster kubectl logs -f rabbitmq-cluster-0 
'/etc/rabbitmq/rabbitmq.conf' -> '/var/lib/rabbitmq/rabbitmq.conf' 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: list of feature flags found: 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] drop_unroutable_metric 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] empty_basic_get_metric 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] implicit_default_bindings 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] quorum_queue 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] virtual_host_metadata 
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: feature flag states written to disk: yes 
2021-02-17 03:30:39.544 [info] <0.269.0> ra: meta data store initialised. 0 record(s) recovered 
2021-02-17 03:30:39.547 [info] <0.274.0> WAL: recovering [] 
2021-02-17 03:31:10.676 [info] <0.313.0> 
 Starting RabbitMQ 3.8.3 on Erlang 22.3.4.1 
 Copyright (c) 2007-2020 Pivotal Software, Inc. 
 Licensed under the MPL 1.1. Website: https://rabbitmq.com 

  ## ## RabbitMQ 3.8.3 
  ## ## 
  ########## Copyright (c) 2007-2020 Pivotal Software, Inc. 
  ###### ## 
  ########## Licensed under the MPL 1.1. Website: https://rabbitmq.com 

  Doc guides: https://rabbitmq.com/documentation.html 
  Support: https://rabbitmq.com/contact.html 
  Tutorials: https://rabbitmq.com/getstarted.html 
  Monitoring: https://rabbitmq.com/monitoring.html 

  Logs: <stdout> 

  Config file(s): /var/lib/rabbitmq/rabbitmq.conf 

  Starting broker...2021-02-17 03:31:10.678 [info] <0.313.0> 
 node : rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local 
 home dir : /var/lib/rabbitmq 
 config file(s) : /var/lib/rabbitmq/rabbitmq.conf 
 cookie hash : H+IQL2spD4MDV4jPi7mMAg== 
 log(s) : <stdout> 
 database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local 
2021-02-17 03:31:10.695 [info] <0.313.0> Running boot step pre_boot defined by app rabbit 

...省略中間內容 

2021-02-17 03:31:13.273 [info] <0.824.0> Statistics database started. 
2021-02-17 03:31:13.273 [info] <0.823.0> Starting worker pool 'management_worker_pool' with 3 processes in it 
2021-02-17 03:31:13.467 [info] <0.9.0> Server startup complete; 5 plugins started. 
 * rabbitmq_peer_discovery_k8s 
 * rabbitmq_management 
 * rabbitmq_web_dispatch 
 * rabbitmq_management_agent 
 * rabbitmq_peer_discovery_common 
 completed with 5 plugins. 
2021-02-17 03:32:30.060 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up 
2021-02-17 03:32:31.264 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up 
2021-02-17 03:33:31.280 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up 
2021-02-17 03:33:32.627 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up

進入到pod中通過用戶端查看集群狀態

➜ rabbitmq-cluster kubectl exec -it rabbitmq-cluster-0 bash 
root@rabbitmq-cluster-0:/# rabbitmqctl cluster_status 
Cluster status of node rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local ... 
Basics 

Cluster name: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local 

Disk Nodes 

rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local 
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local 
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local 

Running Nodes 

rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local 
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local 
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local 

Versions 

rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1 
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1 
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1 

Alarms 

(none) 

Network Partitions 

(none) 

Listeners 

Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication 
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API 
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication 
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API 
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication 
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API 

Feature flags 

Flag: drop_unroutable_metric, state: enabled 
Flag: empty_basic_get_metric, state: enabled 
Flag: implicit_default_bindings, state: enabled 
Flag: quorum_queue, state: enabled 
Flag: virtual_host_metadata, state: enabled

通過NodePort訪問管理介面

kubectl get svc -l app=rabbitmq-cluster

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

rabbitmq-cluster ClusterIP None 5672/TCP 8m47s

rabbitmq-cluster-manage NodePort 10.1.239.191 15672:30888/TCP 8m47s

進k9s看狀態是否為Running

確認pvc持久性檔案已建立

mq新增一個管理員指令

登入mq ui 可以看到集群已建立

Kubernetes 佈署 RabbitMQ

Bytony

By tony

Related Post

Telepresence 在本地除錯 Kubernetes 微服務

k8s 資源管理工具ktop

GKE 安裝 Cilium Service Mesh 1.3.0

發佈留言取消回覆

You missed

Podman 5 攜帶多平台鏡像和虛擬機器支援而來