cluster planning
- k8s-01: 172.17.10.51
- k8s-02: 172.17.10.52
- k8s-03: 172.17.10.53
The three machines are mixed to deploy the etcd, master cluster and woker cluster of this document.
If there is no special instruction, the initialization operation of this document needs to be performed on all nodes
set hostname
hostnamectl set-hostname k8s-01 # Replace k8s-01 with your current hostname
If DNS does not support host name resolution, you also need to add the correspondence between the host name and IP in the /etc/hosts file of each machine:
cat >> /etc/hosts <<EOF
172.17.10.51 k8s-01
172.17.10.52 k8s-02
172.17.10.53 k8s-03
EOF
Use the bash command to take effect
Add node trust relationship
This operation only needs to be performed on the k8s-01 node. Set the root account to log in to all nodes without a password:
ssh-keygen -t rsa
ssh-copy-id k8s-01
ssh-copy-id k8s-02
ssh-copy-id k8s-03
Update the PATH variable
echo 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc
source /root/.bashrc
Install dependencies
yum install -y epel-release
yum install -y chrony conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget socat git
- The kube-proxy in this document uses ipvs mode, and ipvsadm is the management tool of ipvs;
- Each machine in the etcd cluster needs time synchronization, and chrony is used for system time synchronization;
turn off firewall
Close the firewall, clear the firewall rules, and set the default forwarding policy:
systemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
iptables -P FORWARD ACCEPT
Close the swap partition
Close the swap partition, otherwise the kubelet will fail to start (you can set the kubelet startup parameter --fail-swap-on to false to turn off the swap check):
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
Disable SELinux
Turn off SELinux, otherwise the kubelet may report an error Permission denied when mounting the directory:
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
Optimize kernel parameters
cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
net.ipv4.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh3=4096
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
cp kubernetes.conf /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf
Set system time zone
timedatectl set-timezone Asia/Shanghai
Set system clock synchronization
systemctl enable chronyd
systemctl start chronyd
Check sync status:
timedatectl status
output:
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
- System clock synchronized: yes, indicating that the clock is synchronized;
- NTP service: active, indicating that the clock synchronization service is enabled;
# the current UTC Write time to hardware clock
timedatectl set-local-rtc 0
# Restart services that depend on system time
systemctl restart rsyslog
systemctl restart crond
Close irrelevant services
systemctl stop postfix && systemctl disable postfix
Create related directories
Create a directory:
mkdir -p /opt/k8s/{bin,work} /etc/{kubernetes,etcd}/certDistribution cluster configuration parameter script
Subsequent environment variables are defined in the file environment.sh , please modify it according to your own machine and network conditions. Then copy to all nodes
source environment.sh # Modify first
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp environment.sh root@${node_ip}:/opt/k8s/bin/
ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
done
upgrade kernel
There are some Bugs in the 3.10.x kernel that comes with the CentOS 7.x system, which makes the running Docker and Kubernetes unstable, for example:
- The higher version of docker (after 1.13) has enabled the kernel memory account function supported by the 3.10 kernel experiment (it cannot be closed), which will cause cgroup memory leak when the node pressure is high, such as frequently starting and stopping containers;
- Network device reference count leaks, which will cause an error similar to: "kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1";
The solution is as follows:
- Upgrade the kernel to above 4.4.X;
- Or, compile the kernel manually, disable the CONFIG_MEMCG_KMEM feature;
- Alternatively, install Docker 18.09.1 and above which fixes this issue. But since kubelet will also set kmem (it vendor has runc), you need to recompile kubelet and specify GOFLAGS="-tags=nokmem";
git clone --branch v1.14.1 --single-branch --depth 1 https://github.com/kubernetes/kubernetes
cd kubernetes
KUBE_GIT_VERSION=v1.14.1 ./build/run.sh make kubelet GOFLAGS="-tags=nokmem"
Here is a solution to upgrade the kernel:
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
# Check after installation /boot/grub2/grub.cfg Corresponding kernel menuentry Does it contain initrd16 Configure, if not, install it again!
yum --enablerepo=elrepo-kernel install -y kernel-lt
# Set boot to boot from new kernel
grub2-set-default 0
Restart the machine:
sync
reboot