Elasticsearch builds clusters based on docker compose

Building Elasticsearch cluster based on docker compose

1. Preface

1. Preface

Install elastic search cluster based on docker compose, kibana visual component, and monitor cluster information through cerebro tool

Elasticsearch official website: https://www.elastic.co/cn/

Docker

install

yum install -y docker

start-up

systemctl start docker

test

docker --version

Compose

Docker Compose is a docker tool used to define and run complex applications. An application using docker container is usually composed of multiple containers. Using Docker Compose eliminates the need for shell scripts to start containers.
Compose manages multiple Docker containers through a configuration file. In the configuration file, all containers are defined through services, and then the Docker compose script is used to start, stop and restart the application, the services in the application and all containers that depend on services. It is very suitable for the scenario of combining multiple containers for development.

install

 sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Apply executable permissions to binaries:

sudo chmod +x /usr/local/bin/docker-compose

test

docker-compose --version

Compose and Docker compatibility

compose file format version docker version
3.4 17.09.0+
3.3 17.06.0+
3.2 17.04.0+
3.1 1.13.1+
3.0 1.13.0+
2.3 17.06.0+
2.2 1.13.0+
2.1 1.12.0+
2.0 1.10.0+
1.0 1.9.1.+

configuration file

Official documents

Create docker compose elasticsearch yaml

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic
  kibana:
    image: docker.elastic.co/kibana/kibana:7.15.2
    container_name: kibana
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    depends_on:
      - es01
    networks:
      - elastic
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

elasticsearch/config/elasticsearch.yml

network.host: 0.0.0.0
http.port: 9200
# Enable es cross domain
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
# Turn on safety control
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12

kibana/config/kibana.yml

server.name: kibana
server.host: "0.0.0.0"
xpack.monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: "elastic"  # es account
elasticsearch.password: "*******"   # es password enter the password set by the container
i18n.locale: zh-CN # chinese

Upload and run

Upload the file docker compose elasticsearch Yaml to the directory you created

elasticsearch security policy

Generate certificate
Execute commands in sequence
1. Create a temporary container
2. Enter the container
3. Create ca [enter directly without entering password]
4. Create certificate [enter directly without entering password]
5. Exit the container and copy the certificate in the container
6. Delete this temporary container

docker run -d  docker.elastic.co/elasticsearch/elasticsearch:7.15.2 --name=es
docker exec -it es /bin/bash
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
docker cp es:/usr/share/elasticsearch/elastic-certificates.p12 .
docker rm -f es

If the password is entered when creating the certificate, it needs to be executed in the container, otherwise the certificate will fail to pass the authentication when the cluster starts.
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

Start all containers

docker-compose -f docker-compose-elasticsearch.yaml up -d

Enter the container to change the password

docker exec -it es01 /bin/bash

[root@localhost elasticsearch-cluster]# docker exec -it es01 /bin/bash
bash-4.4# ./bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana_system]: 
Reenter password for [kibana_system]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

By command docker-compose -f docker-compose-elasticsearch.yaml up -d function

```bash
[root@localhost elasticsearch-cluster]# docker-compose -f docker-compose-elasticsearch.yaml up -d
Creating network "elasticsearch-cluster_elastic" with driver "bridge"
Creating volume "elasticsearch-cluster_data01" with local driver
Creating volume "elasticsearch-cluster_data02" with local driver
Creating volume "elasticsearch-cluster_data03" with local driver
Pulling es01 (docker.elastic.co/elasticsearch/elasticsearch:7.15.2)...
Trying to pull repository docker.elastic.co/elasticsearch/elasticsearch ... 
7.15.2: Pulling from docker.elastic.co/elasticsearch/elasticsearch
009c11f4ddee: Pull complete
8772b99d888d: Pull complete
bd8b744bf3bf: Pull complete
2a41be2c565a: Pull complete
e7e9200dd33e: Pull complete
Digest: sha256:a1dce08d504b22e87adc849c94dcae53f6a0bd12648a4d99d7f9fc07bb2e8a3e
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.15.2
Creating es02 ... done
Creating es01 ... done
Creating es03 ... done

View the running containers through the docker ps command

[root@localhost elasticsearch-cluster]# docker ps
CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
f43b017dd23a        docker.elastic.co/elasticsearch/elasticsearch:7.15.2   "/bin/tini -- /usr..."   17 seconds ago      Up 12 seconds       9200/tcp, 9300/tcp                 es03
7ed565d7eb4e        docker.elastic.co/elasticsearch/elasticsearch:7.15.2   "/bin/tini -- /usr..."   17 seconds ago      Up 12 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   es01
fb89e106eea2        docker.elastic.co/elasticsearch/elasticsearch:7.15.2   "/bin/tini -- /usr..."   17 seconds ago      Up 12 seconds       9200/tcp, 9300/tcp                 es02

View the operation log of the container through docker logs -f es01

[root@localhost elasticsearch-cluster]# docker logs -f es01
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Elasticsearch (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.2.jar)
WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Security (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.2.jar)
WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
{"type": "server", "timestamp": "2021-12-30T04:49:03,608Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "version[7.15.2], pid[6], build[default/docker/93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c/2021-11-04T14:04:42.515624022Z], OS[Linux/3.10.0-1160.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]" }
..........
..........
..........

Open firewall port: 92009300

[root@localhost elasticsearch-cluster]# firewall-cmd --zone=public --add-port=9200/tcp --add-port=9300/tcp --permanent&&firewall-cmd --reload
success
success
[root@localhost elasticsearch-cluster]# firewall-cmd --list-ports
6379/tcp 9200/tcp 9300/tcp
[root@localhost elasticsearch-cluster]#

Visit Kibana

View node

[root@localhost elasticsearch-cluster]# curl -X GET "localhost:9200/_cat/nodes?v=true&pretty"
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.18.0.4           64          89   6    0.11    0.18     0.19 cdfhilmrstw *      es02
172.18.0.3           68          89   6    0.11    0.18     0.19 cdfhilmrstw -      es01
172.18.0.2           53          89   6    0.11    0.18     0.19 cdfhilmrstw -      es03
[root@localhost elasticsearch-cluster]#