Building Elasticsearch cluster based on docker compose
1. Preface
Install elastic search cluster based on docker compose, kibana visual component, and monitor cluster information through cerebro tool
Elasticsearch official website: https://www.elastic.co/cn/
Docker
install
yum install -y docker
start-up
systemctl start docker
test
docker --version
Compose
Docker Compose is a docker tool used to define and run complex applications. An application using docker container is usually composed of multiple containers. Using Docker Compose eliminates the need for shell scripts to start containers.
Compose manages multiple Docker containers through a configuration file. In the configuration file, all containers are defined through services, and then the Docker compose script is used to start, stop and restart the application, the services in the application and all containers that depend on services. It is very suitable for the scenario of combining multiple containers for development.
install
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
Apply executable permissions to binaries:
sudo chmod +x /usr/local/bin/docker-compose
test
docker-compose --version
Compose and Docker compatibility
compose file format version docker version
3.4 17.09.0+
3.3 17.06.0+
3.2 17.04.0+
3.1 1.13.1+
3.0 1.13.0+
2.3 17.06.0+
2.2 1.13.0+
2.1 1.12.0+
2.0 1.10.0+
1.0 1.9.1.+
configuration file
Create docker compose elasticsearch yaml
version: '2.2' services: es01: image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2 container_name: es01 environment: - node.name=es01 - cluster.name=es-docker-cluster - discovery.seed_hosts=es02,es03 - cluster.initial_master_nodes=es01,es02,es03 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 volumes: - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - ./elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12 - data01:/usr/share/elasticsearch/data ports: - 9200:9200 networks: - elastic es02: image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2 container_name: es02 environment: - node.name=es02 - cluster.name=es-docker-cluster - discovery.seed_hosts=es01,es03 - cluster.initial_master_nodes=es01,es02,es03 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 volumes: - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - ./elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12 - data02:/usr/share/elasticsearch/data networks: - elastic es03: image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2 container_name: es03 environment: - node.name=es03 - cluster.name=es-docker-cluster - discovery.seed_hosts=es01,es02 - cluster.initial_master_nodes=es01,es02,es03 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 volumes: - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - ./elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12 - data03:/usr/share/elasticsearch/data networks: - elastic kibana: image: docker.elastic.co/kibana/kibana:7.15.2 container_name: kibana volumes: - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml ports: - 5601:5601 environment: ELASTICSEARCH_URL: http://es01:9200 ELASTICSEARCH_HOSTS: http://es01:9200 depends_on: - es01 networks: - elastic volumes: data01: driver: local data02: driver: local data03: driver: local networks: elastic: driver: bridge
elasticsearch/config/elasticsearch.yml
network.host: 0.0.0.0 http.port: 9200 # Enable es cross domain http.cors.enabled: true http.cors.allow-origin: "*" http.cors.allow-headers: Authorization # Turn on safety control xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
kibana/config/kibana.yml
server.name: kibana server.host: "0.0.0.0" xpack.monitoring.ui.container.elasticsearch.enabled: true elasticsearch.username: "elastic" # es account elasticsearch.password: "*******" # es password enter the password set by the container i18n.locale: zh-CN # chinese
Upload and run
Upload the file docker compose elasticsearch Yaml to the directory you created
elasticsearch security policy
Generate certificate
Execute commands in sequence
1. Create a temporary container
2. Enter the container
3. Create ca [enter directly without entering password]
4. Create certificate [enter directly without entering password]
5. Exit the container and copy the certificate in the container
6. Delete this temporary container
docker run -d docker.elastic.co/elasticsearch/elasticsearch:7.15.2 --name=es docker exec -it es /bin/bash ./bin/elasticsearch-certutil ca ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 docker cp es:/usr/share/elasticsearch/elastic-certificates.p12 . docker rm -f es
If the password is entered when creating the certificate, it needs to be executed in the container, otherwise the certificate will fail to pass the authentication when the cluster starts.
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
Start all containers
docker-compose -f docker-compose-elasticsearch.yaml up -d
Enter the container to change the password
docker exec -it es01 /bin/bash
[root@localhost elasticsearch-cluster]# docker exec -it es01 /bin/bash bash-4.4# ./bin/elasticsearch-setup-passwords interactive Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana_system]: Reenter password for [kibana_system]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Changed password for user [apm_system] Changed password for user [kibana_system] Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [beats_system] Changed password for user [remote_monitoring_user] Changed password for user [elastic] By command docker-compose -f docker-compose-elasticsearch.yaml up -d function ```bash [root@localhost elasticsearch-cluster]# docker-compose -f docker-compose-elasticsearch.yaml up -d Creating network "elasticsearch-cluster_elastic" with driver "bridge" Creating volume "elasticsearch-cluster_data01" with local driver Creating volume "elasticsearch-cluster_data02" with local driver Creating volume "elasticsearch-cluster_data03" with local driver Pulling es01 (docker.elastic.co/elasticsearch/elasticsearch:7.15.2)... Trying to pull repository docker.elastic.co/elasticsearch/elasticsearch ... 7.15.2: Pulling from docker.elastic.co/elasticsearch/elasticsearch 009c11f4ddee: Pull complete 8772b99d888d: Pull complete bd8b744bf3bf: Pull complete 2a41be2c565a: Pull complete e7e9200dd33e: Pull complete Digest: sha256:a1dce08d504b22e87adc849c94dcae53f6a0bd12648a4d99d7f9fc07bb2e8a3e Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.15.2 Creating es02 ... done Creating es01 ... done Creating es03 ... done
View the running containers through the docker ps command
[root@localhost elasticsearch-cluster]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f43b017dd23a docker.elastic.co/elasticsearch/elasticsearch:7.15.2 "/bin/tini -- /usr..." 17 seconds ago Up 12 seconds 9200/tcp, 9300/tcp es03 7ed565d7eb4e docker.elastic.co/elasticsearch/elasticsearch:7.15.2 "/bin/tini -- /usr..." 17 seconds ago Up 12 seconds 0.0.0.0:9200->9200/tcp, 9300/tcp es01 fb89e106eea2 docker.elastic.co/elasticsearch/elasticsearch:7.15.2 "/bin/tini -- /usr..." 17 seconds ago Up 12 seconds 9200/tcp, 9300/tcp es02
View the operation log of the container through docker logs -f es01
[root@localhost elasticsearch-cluster]# docker logs -f es01 WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Elasticsearch (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.2.jar) WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Security (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.2.jar) WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release {"type": "server", "timestamp": "2021-12-30T04:49:03,608Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "version[7.15.2], pid[6], build[default/docker/93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c/2021-11-04T14:04:42.515624022Z], OS[Linux/3.10.0-1160.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]" } .......... .......... ..........
Open firewall port: 92009300
[root@localhost elasticsearch-cluster]# firewall-cmd --zone=public --add-port=9200/tcp --add-port=9300/tcp --permanent&&firewall-cmd --reload success success [root@localhost elasticsearch-cluster]# firewall-cmd --list-ports 6379/tcp 9200/tcp 9300/tcp [root@localhost elasticsearch-cluster]#
Visit Kibana
View node
[root@localhost elasticsearch-cluster]# curl -X GET "localhost:9200/_cat/nodes?v=true&pretty" ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 172.18.0.4 64 89 6 0.11 0.18 0.19 cdfhilmrstw * es02 172.18.0.3 68 89 6 0.11 0.18 0.19 cdfhilmrstw - es01 172.18.0.2 53 89 6 0.11 0.18 0.19 cdfhilmrstw - es03 [root@localhost elasticsearch-cluster]#
Run Cerebro
Download address: https://github.com/lmenezes/cerebro/releases
Unzip and run: cerebro bat
Modify conf/application es password in conf
Double click cerebro Bat run