With the increase of projects deployed on k8s clusters, the server cpu and memory have reached the bottleneck and often get stuck. When it is necessary to expand the cluster capacity, it is found that the company has five 16 core 32G idle servers, which can access the external network, but there is no public ip. They are all NAT Intranet environment, and http is also blocked by ISP.
The existing cluster is deployed on Tencent cloud and has a public network environment, so let the master node act as the gateway and route to create a virtual local area network (vpn) to complete the interconnection, and let the company's servers join the cluster.
This is a simplified topology

1, Environment preparation: CentOS 7 is installed on all three hosts, and the experimental kernel version is 3.10.0-1160.15.2 el7. x86_ sixty-four

1. Turn off the firewall of all hosts and update the yum source
    

systemctl stop firewalld
systemctl disable firewalld
yum update -y

2. Set the host name separately. It is troublesome to set the host name after configuring the network

 hostnamectl set-hostname master
 hostnamectl set-hostname node1
 hostnamectl set-hostname node2

2, Install vpn service on public network server

1. Install l2tp and ipsec services on the master node directly with one click of the script. The script will ask PSK, user name and password
 

chmod +x l2tp.sh
./l2tp.sh
####Prompt after successful installation############
Server IP: 1.15.132.93
PSK      : xxxxx
Username : xxxxx
Password : xxxxx
############################

2. Start l2tp and ipsec. If the firewall is not closed in the production environment, you need to open ports 500, 4500 and 1701
 

systemctl start ipsec
systemctl start xl2tpd
systemctl enable ipsec xl2tpd

3. Add a remote connection account, add two node accounts for the server to join the virtual LAN, and an admin account for the user to remotely manage the intranet server.

l2tp -a
####Account & Password ######
l2tp -l
#######Show account#########
+-------------------------------------------+
|            Username |            Password |
+-------------------------------------------+
|               node1 |          xxxxxxxxxx |
|               node2 |          xxxxxxxxxx |
|               admin |          xxxxxxxxxx |
+-------------------------------------------+

4. Assign fixed IP to two node accounts. If no fixed IP is assigned, the node will join the virtual LAN at random every time, which is inconvenient for k8s cluster management
 

vi /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client    server    secret    IP addresses
node1    l2tpd    xxxxxxxxxx       192.168.18.10
node2    l2tpd    xxxxxxxxxx       192.168.18.11
admin    l2tpd    xxxxxxxxxx       *

2, Connect vpn to intranet server and join virtual LAN

1. Installing clients on node1 and node2

yum install -y epel-release
yum -y install xl2tpd ppp

2. Modify the configuration file and start xl2tpd. Different servers use different accounts.

vi /etc/xl2tpd/xl2tpd.conf

#################Complete content##################
[lac myvpn]
name = node1
lns = 1.15.132.93
pppoptfile = /etc/ppp/peers/myvpn.xl2tpd
ppp debug = no
#######################################

vi /etc/ppp/peers/myvpn.xl2tpd

#################Complete content##################
remotename myvpn
user "node1"
password "node1 Password for"
unit 0
debug
kdebug 1
mtu 1000
nobsdcomp
nodeflate
noaccomp
nopcomp
novj
#######################################

vi /etc/ipsec.secrets
##################Complete content#################
%any 1.15.132.93 : PSK "The server PSK"
##########################################

vi /etc/ppp/options.xl2tpd

################Add content###################
lcp-echo-interval 300
lcp-echo-failure 10
##########################################

systemctl start xl2tpd
systemctl enable xl2tpd

3. Dial up connection

sh -c 'echo "c myvpn" > /var/run/xl2tpd/l2tp-control'

4. Test whether the virtual LAN is connected
① ifconfig checks the network card information to see if the connection is successful. If there is no ppp network card, use tail / var / log / messages to check the failure reason
   

② Check whether the client and gateway are unobstructed

③ Check whether the clients are interconnected
 

There is no ping communication between clients. Use tracroute to track the route
 

The reason is that the default gateway is used and a static route needs to be added. The gateway specified for the 192.168.18.0/24 network segment is 192.168.18.1. The same node node needs to access master eth0 # 192.168.1.50 and also needs to add a static route

route add -net 192.168.18.0/24 gw 192.168.18.1
route add -host 192.168.1.50 gw 192.168.18.1

At this point, the network is connected and the deployment is completed
 

④ Write shell script to realize automatic connection after startup and reconnection after disconnection

vi /root/reconnect.sh
#####################################################################
#!/bin/bash
IP=192.168.18.1
while true  
do  
    ping -c 5 -q $IP > /dev/null  
    if [ $? -ne 0 ]; then  
		sh -c 'echo "c myvpn" > /var/run/xl2tpd/l2tp-control'
        sleep 10
        route add -net 192.168.18.0/24 gw 192.168.18.1
		route add -host 192.168.1.50 gw 192.168.18.1 
    fi  
    sleep 30  
done  
####################################################################

vi /etc/rc.d/rc.local

#########################Add at the end####################################
sh /root/reconnect.sh &
#####################################################################

chmod +x /root/reconnect.sh
chmod +x /etc/rc.d/rc.local

After reboot, wait for about 30 seconds. If the network is normal, the representative can reconnect