K8S initialization system and global variables

cluster planning

  • k8s-01:
  • k8s-02:
  • k8s-03:

The three machines are mixed to deploy the etcd, master cluster and woker cluster of this document.

If there is no special instruction, the initialization operation of this document needs to be performed on all nodes

set hostname

hostnamectl set-hostname k8s-01 # Replace k8s-01 with your current hostname

If DNS does not support host name resolution, you also need to add the correspondence between the host name and IP in the /etc/hosts​​ file of each machine:

cat >> /etc/hosts <<EOF k8s-01 k8s-02 k8s-03

Use the bash command to take effect

Add node trust relationship

This operation only needs to be performed on the k8s-01 node. Set the root account to log in to all nodes without a password:

ssh-keygen -t rsa 
ssh-copy-id k8s-01
ssh-copy-id k8s-02
ssh-copy-id k8s-03

Update the PATH variable

echo 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc
source /root/.bashrc

Install dependencies

yum install -y epel-release
yum install -y chrony conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget socat git
  • The kube-proxy in this document uses ipvs mode, and ipvsadm is the management tool of ipvs;
  • Each machine in the etcd cluster needs time synchronization, and chrony is used for system time synchronization;

turn off firewall

Close the firewall, clear the firewall rules, and set the default forwarding policy:

systemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat

Close the swap partition

Close the swap partition, otherwise the kubelet will fail to start (you can set the kubelet startup parameter --fail-swap-on to false to turn off the swap check):

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Disable SELinux

Turn off SELinux, otherwise the kubelet may report an error Permission denied when mounting the directory:

setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

Optimize kernel parameters

cat > kubernetes.conf <<EOF
cp kubernetes.conf /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf

Set system time zone

timedatectl set-timezone Asia/Shanghai

Set system clock synchronization

systemctl enable chronyd
systemctl start chronyd

Check sync status:

timedatectl status


System clock synchronized: yes
NTP service: active
RTC in local TZ: no
  • ​​​​System clock synchronized: yes​, indicating that the clock is synchronized;
  • ​​​​NTP service: active​​, indicating that the clock synchronization service is enabled;
# the current UTC Write time to hardware clock
timedatectl set-local-rtc 0

# Restart services that depend on system time
systemctl restart rsyslog
systemctl restart crond

Close irrelevant services

systemctl stop postfix && systemctl disable postfix

Create related directories

Create a directory:

mkdir -p /opt/k8s/{bin,work} /etc/{kubernetes,etcd}/cert

Distribution cluster configuration parameter script

Subsequent environment variables are defined in the file​​ ​environment.sh​ ​, please modify it according to your own machine and network conditions. Then copy to all nodes

source environment.sh # Modify first
for node_ip in ${NODE_IPS[@]}
echo ">>> ${node_ip}"
scp environment.sh root@${node_ip}:/opt/k8s/bin/
ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"

upgrade kernel

There are some Bugs in the 3.10.x kernel that comes with the CentOS 7.x system, which makes the running Docker and Kubernetes unstable, for example:

  1. The higher version of docker (after 1.13) has enabled the kernel memory account function supported by the 3.10 kernel experiment (it cannot be closed), which will cause cgroup memory leak when the node pressure is high, such as frequently starting and stopping containers;
  2. Network device reference count leaks, which will cause an error similar to: "kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1";

The solution is as follows:

  1. Upgrade the kernel to above 4.4.X;
  2. Or, compile the kernel manually, disable the CONFIG_MEMCG_KMEM feature;
  3. Alternatively, install Docker 18.09.1 ​​and above which fixes this issue. But since kubelet will also set kmem (it vendor has runc), you need to recompile kubelet and specify GOFLAGS="-tags=nokmem";
git clone --branch v1.14.1 --single-branch --depth 1 https://github.com/kubernetes/kubernetes
cd kubernetes
KUBE_GIT_VERSION=v1.14.1 ./build/run.sh make kubelet GOFLAGS="-tags=nokmem"

Here is a solution to upgrade the kernel:

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
# Check after installation /boot/grub2/grub.cfg Corresponding kernel menuentry Does it contain initrd16 Configure, if not, install it again!
yum --enablerepo=elrepo-kernel install -y kernel-lt
# Set boot to boot from new kernel
grub2-set-default 0

​Restart the machine:


Tags: Kubernetes Operation & Maintenance ssh Container Cloud Native

Posted by rookie on Mon, 23 Jan 2023 17:12:36 +1030