Manually deploy rabbitmq cluster based on k8s

catalogue

1. RabbitMQ introduction

RabbitMQ is an open source message broker software (also known as message oriented middleware) that implements the advanced message queuing protocol AMQP. RabbitMQ server is written in Erlang language, and clustering and failover are built on the framework of open telecommunications platform. AMQP: Advanced Message Queue, Advanced Message Queue protocol. It is an open standard of application layer protocol. It is designed for message oriented middleware. The client and message oriented middleware based on this protocol can transmit messages, which is not limited by the conditions of products and development languages

AMQP has the following characteristics:

  • Reliability: some mechanisms are used to ensure reliability, such as persistence, transmission confirmation and release confirmation
  • Flexible Routing: route messages through Exchange before they enter the queue. For typical routing functions, Rabbit has provided some built-in Exchange to implement. For more complex routing functions, multiple exchanges can be bound together, and their own Exchange can be realized through plug-in mechanism
  • Message Clustering: multiple RabbitMQ servers can form a cluster to form a logical Broker
  • Highly available queues: the queue can be mirrored on the machines in the cluster, so that the queue can still be used in case of problems in some nodes
  • Multi protocol: supports multiple message queuing protocols, such as STOMP, MQTT, etc
  • Multilingual clients: almost all common languages are supported, such as Java and NET, Ruby, etc
  • Management UI: provides an easy-to-use user interface that enables users to monitor and manage many aspects of message Broker
  • Tracing mechanism: if the message is abnormal, RabbitMQ provides a message tracing mechanism, and the user can find out what happened
  • Plug in system: it provides many plug-ins to extend from many aspects, and you can also edit your own plug-ins

2. Persistence and mirroring queues for RabbitMQ

RabbitMQ persistence is divided into Exchange, Queue and Message

  • Exchange and Queue persistence: refers to the persistence of exchange and Queue metadata. The persistence is itself. When the service is down, exchange and Queue will not exist
  • Message persistence: just as the name suggests, it is to persist every message body. When the service is down, the message will not be lost

The Queue image of RabbitMQ means that the master node will synchronize to other nodes after receiving the request, so as to ensure high availability. In confirm mode, the specific process is as follows

clientpublisher send message –> master node Received a message –> master node Persist messages to disk –> Send messages to other nodes asynchronously –> master take ack Return to client publisher

3. Deployment of RabbitMQ cluster in k8s

Deploy RabbitMQ in k8s in the form of cluster, on the premise that each node of RabbitMQ can perform mutual service discovery like the traditional way. Therefore, RabbitMQ passes RabbitMQ in the k8s cluster_ peer_ discovery_ K8s plugin interacts with k8s apiserver to obtain the URL of each service, and RabbitMQ must be matched with statefullset and headless service in k8s cluster

It should be noted that rabbitmq_peer_discovery_k8s is developed by rabbitmq based on rabbitmq autocluster, an official third-party open source project The peer discovery plug-in under Kubernetes provided by version x and above can realize the automatic deployment of rabbitmq cluster in k8s, so it is lower than 3.7 For version x, please use rabbitmq autocluster

3.1 introduction to environment

The version deployed in this article is 3.8.3

The default deployment is under the default namespace,

Persistent storage is storageclass dynamic storage, and the underlying layer is nfs

Image address rabbitmq:3.8.3-management

From official yaml https://github.com/rabbitmq/diy-kubernetes-examples

3.2 creating configmap

01-rabbitmq-configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: rabbitmq-cluster-config
  namespace: default
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
    enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
    rabbitmq.conf: |
      default_user = admin
      default_pass = 123!@#
      ## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more.
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address?
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible.
      ## Set to "hostname" to use pod hostnames.
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME
      ## environment variable.
      cluster_formation.k8s.address_type = hostname
      ## How often should node cleanup checks run?
      cluster_formation.node_cleanup.interval = 30
      ## Set to false if automatic removal of unknown/absent nodes
      ## is desired. This can be dangerous, see
      ##  * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup
      ##  * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## See https://www.rabbitmq.com/ha.html#master-migration-data-locality
      queue_master_locator=min-masters
      ## See https://www.rabbitmq.com/access-control.html#loopback-users
      loopback_users.guest = false
      cluster_formation.randomized_startup_delay_range.min = 0
      cluster_formation.randomized_startup_delay_range.max = 2
      # default is rabbitmq-cluster's namespace
      # hostname_suffix
      cluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.default.svc.cluster.local
      # memory
      vm_memory_high_watermark.absolute = 1GB
      # disk
      disk_free_limit.absolute = 2GB

Description of some parameters:

  • enabled_plugins: the name of the plug-in declared open
  • default_pass/default_pass: declare the user name and password (although some article records can be declared through environment variables, after testing, if configmap, rabbitmq configuration file, is specified for this version, the declared environment variables are useless and need to be specified in the configuration file)
  • cluster_formation.k8s.address_type: calculate the peer-to-peer node list from the pod container list returned by k8s. Here, only the host name can be used. In the official example, ip is used. However, by default, the ip of pod in k8s is not fixed, which may lead to the loss of node configuration and data. The host name of pod will be fixed by referencing metadata in the subsequent yaml.

For more parameters, please refer to the official documents, which will not be repeated here

3.3 creating a service

02-rabbitmq-service.yaml

kind: Service
apiVersion: v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-cluster
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: rmqport
    port: 5672
    targetPort: 5672
  selector:
    app: rabbitmq-cluster

---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-cluster-manage
  namespace: default
spec:
  ports:
  - name: http
    port: 15672
    protocol: TCP
    targetPort: 15672
  selector:
    app: rabbitmq-cluster
  type: NodePort

Two services are defined above. One is the Service port of rabbitmq and the other is the port of the management interface. Users can access them externally. Here, they are exposed through NodePort

3.4 creating rbac authorization

The previous introduction mentioned that RabbitMQ interacts with k8s apiserver through the plug-in to obtain the relevant information of nodes in the cluster, so it needs RBAC authorization

03-rabbitmq-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq-cluster
  namespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rabbitmq-cluster
  namespace: default
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rabbitmq-cluster
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rabbitmq-cluster
subjects:
- kind: ServiceAccount
  name: rabbitmq-cluster
  namespace: default

3.5 create statefullset

RabbitMQ is deployed as a stateful application in k8s, so the controller type is StatefulSet, and pvc related content is also defined in yaml

04-rabbitmq-cluster-sts.yaml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-cluster
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq-cluster
  serviceName: rabbitmq-cluster
  template:
    metadata:
      labels:
        app: rabbitmq-cluster
    spec:
      containers:
      - args:
        - -c
        - cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.sh
          rabbitmq-server
        command:
        - sh
        env:
        - name: TZ
          value: 'Asia/Shanghai'
        - name: RABBITMQ_ERLANG_COOKIE
          value: 'SWvCP0Hrqv43NG7GybHC95ntCJKoW8UyNFWnBEWG8TY='
        - name: K8S_SERVICE_NAME
          value: rabbitmq-cluster
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_CONFIG_FILE
          value: /var/lib/rabbitmq/rabbitmq.conf
        image: rabbitmq:3.8.3-management
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - status
          # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 15
        name: rabbitmq
        ports:
        - containerPort: 15672
          name: http
          protocol: TCP
        - containerPort: 5672
          name: amqp
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - status
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        volumeMounts:
        - mountPath: /etc/rabbitmq
          name: config-volume
          readOnly: false
        - mountPath: /var/lib/rabbitmq
          name: rabbitmq-storage
          readOnly: false
        - name: timezone
          mountPath: /etc/localtime
          readOnly: true
      serviceAccountName: rabbitmq-cluster
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config-volume
        configMap:
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf
          - key: enabled_plugins
            path: enabled_plugins
          name: rabbitmq-cluster-config
      - name: timezone
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-storage
    spec:
      accessModes:
      - ReadWriteMany
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 2Gi

3.6 deployment inspection

➜  rabbitmq-cluster pwd
/Users/ssgeek/Documents/k8s-manifests/rabbitmq-cluster
➜  rabbitmq-cluster ls
01-rabbitmq-configmap.yaml   02-rabbitmq-service.yaml     03-rabbitmq-rbac.yaml        04-rabbitmq-cluster-sts.yaml
➜  rabbitmq-cluster kubectl apply -f .
configmap/rabbitmq-cluster-config created
service/rabbitmq-cluster created
service/rabbitmq-cluster-manage created
serviceaccount/rabbitmq-cluster created
role.rbac.authorization.k8s.io/rabbitmq-cluster created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster created
statefulset.apps/rabbitmq-cluster created

Wait for a while to view the relevant resources created

➜  rabbitmq-cluster kubectl get po,sts -l app=rabbitmq-cluster
NAME                     READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-0   1/1     Running   0          3m22s
pod/rabbitmq-cluster-1   1/1     Running   0          2m18s
pod/rabbitmq-cluster-2   1/1     Running   0          2m23s

NAME                                READY   AGE
statefulset.apps/rabbitmq-cluster   3/3     3m25s

View the log and observe the status of cluster establishment from the last part of the log

➜  rabbitmq-cluster kubectl logs -f rabbitmq-cluster-0
'/etc/rabbitmq/rabbitmq.conf' -> '/var/lib/rabbitmq/rabbitmq.conf'
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: list of feature flags found:
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] drop_unroutable_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] empty_basic_get_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] implicit_default_bindings
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] quorum_queue
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] virtual_host_metadata
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: feature flag states written to disk: yes
2021-02-17 03:30:39.544 [info] <0.269.0> ra: meta data store initialised. 0 record(s) recovered
2021-02-17 03:30:39.547 [info] <0.274.0> WAL: recovering []
2021-02-17 03:31:10.676 [info] <0.313.0> 
 Starting RabbitMQ 3.8.3 on Erlang 22.3.4.1
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL 1.1. Website: https://rabbitmq.com

  ##  ##      RabbitMQ 3.8.3
  ##  ##
  ##########  Copyright (c) 2007-2020 Pivotal Software, Inc.
  ######  ##
  ##########  Licensed under the MPL 1.1. Website: https://rabbitmq.com

  Doc guides: https://rabbitmq.com/documentation.html
  Support:    https://rabbitmq.com/contact.html
  Tutorials:  https://rabbitmq.com/getstarted.html
  Monitoring: https://rabbitmq.com/monitoring.html

  Logs: <stdout>

  Config file(s): /var/lib/rabbitmq/rabbitmq.conf

  Starting broker...2021-02-17 03:31:10.678 [info] <0.313.0> 
 node           : rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
 home dir       : /var/lib/rabbitmq
 config file(s) : /var/lib/rabbitmq/rabbitmq.conf
 cookie hash    : H+IQL2spD4MDV4jPi7mMAg==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
2021-02-17 03:31:10.695 [info] <0.313.0> Running boot step pre_boot defined by app rabbit

...Omit intermediate content

2021-02-17 03:31:13.273 [info] <0.824.0> Statistics database started.
2021-02-17 03:31:13.273 [info] <0.823.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-02-17 03:31:13.467 [info] <0.9.0> Server startup complete; 5 plugins started.
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
 * rabbitmq_peer_discovery_common
 completed with 5 plugins.
2021-02-17 03:32:30.060 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:32:31.264 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:33:31.280 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:33:32.627 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up

Enter the pod and view the cluster status through the client

➜  rabbitmq-cluster kubectl exec -it rabbitmq-cluster-0 bash
root@rabbitmq-cluster-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local ...
Basics

Cluster name: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local

Disk Nodes

rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local

Running Nodes

rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local

Versions

rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API

Feature flags

Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled

Access the management interface through NodePort

kubectl get svc -l app=rabbitmq-cluster
NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
rabbitmq-cluster          ClusterIP   None           <none>        5672/TCP          8m47s
rabbitmq-cluster-manage   NodePort    10.1.239.191   <none>        15672:30888/TCP   8m47s

Here, manually deploy a RabbitMQ cluster in k8s~

reference resources:
https://www.rabbitmq.com/cluster-formation.html
https://github.com/rabbitmq/diy-kubernetes-examples

Tags: Kubernetes

Posted by jonabomer on Sun, 17 Apr 2022 06:50:42 +0930

Popular Tags

©2023 Programs Wiki