How to perform rolling upgrade of a service in docker swarm?


You must have heard of rolling upgrades. For example, there is a service running multiple instances. If you want to upgrade the service (for example, replace the image), what should you do?

 

Let's take a look at the next part.

 

In this article, a scenario for doing a rolling upgrade is:

 

deploy one service,this service use redis 3.0.6 mirror, then use redis 3.0.7 version image to upgrade

 

 

ok, start.

 

1. Log in to the host of the manager node

2. Deploy the redis service (configure the update delay parameter: 10s)

 

docker service create \
  --replicas 3 \
  --name redis \
  --update-delay 10s \
  172.20.58.152/middleware/redis:3.0.6

 

 

The redis:3.0.6 image is used here

 

[root@nccztsjb-node-07 ~]# docker service create \
>   --replicas 3 \
>   --name redis \
>   --update-delay 10s \
>   172.20.58.152/middleware/redis:3.0.6
c961xuuojwons7gwv923zktmn
overall progress: 3 out of 3 tasks 
1/3: running   [==================================================>] 
2/3: running   [==================================================>] 
3/3: running   [==================================================>] 
verify: Service converged 
[root@nccztsjb-node-07 ~]# docker service ls
ID             NAME      MODE         REPLICAS   IMAGE                                  PORTS
c961xuuojwon   redis     replicated   3/3        172.20.58.152/middleware/redis:3.0.6   
[root@nccztsjb-node-07 ~]# 
[root@nccztsjb-node-07 ~]# docker service ps redis
ID             NAME      IMAGE                                  NODE               DESIRED STATE   CURRENT STATE            ERROR     PORTS
t8e21ufkp984   redis.1   172.20.58.152/middleware/redis:3.0.6   nccztsjb-node-10   Running         Running 35 seconds ago             
bik9ct1yg3xo   redis.2   172.20.58.152/middleware/redis:3.0.6   nccztsjb-node-07   Running         Running 35 seconds ago             
pxw9ihz4bgqm   redis.3   172.20.58.152/middleware/redis:3.0.6   nccztsjb-node-08   Running         Running 35 seconds ago             
[root@nccztsjb-node-07 ~]#

 

 

The service is running normally.

 

#Rolling update policy configuration instructions#

 

  • The rolling update strategy can be configured when the service is deployed
  • --update-delay sets the update delay between each task (how long after the first task runs correctly and the second task starts to update), and units can set s,m,h [e.g., 10m 30 s means 10min 30seconds delay]
  • By default, the scheduler updates one task at a time. You can use the --update-parallelism parameter to set the scheduler to update the number of tasks at the same time.
  • When performing rolling upgrades, when a task returns to the RUNNING state, the scheduler schedules the next task, in this way, until all tasks are updated.
  • When the update process, any task returns FAILED, the scheduler will terminate the update. The --update-failure-action parameter can be used to control the behavior when the update fails.

 

View details of serivce

 

[root@nccztsjb-node-07 ~]# docker service inspect --pretty redis

ID:        c961xuuojwons7gwv923zktmn
Name:        redis
Service Mode:    Replicated
 Replicas:    3
Placement:
UpdateConfig:
 Parallelism:    1
 Delay:        10s
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:    1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:        172.20.58.152/middleware/redis:3.0.6@sha256:d161fc7de1183ed220e28cf5e4cc50f1b535d580f2c3503c0b4ae4f2d63c1b1f
 Init:        false
Resources:
Endpoint Mode:    vip

[root@nccztsjb-node-07 ~]# 

 

 

The above information, focusing on the update configuration UpdateConfig

 

UpdateConfig:

Parallelism: 1 #Number of parallel update instances

Delay: 10s #task update delay

On failure: pause #Behavior when update fails

Monitoring Period: 5s #Monitoring period

Max failure ratio: 0

Update order: stop-first #Update order, stop the task first, then update the task

 

 

3. Use the redis 3.0.7 version image to update the redis service

 

The entire update process will be updated according to the update policy updateconfig

 

docker service update --image 172.20.58.152/middleware/redis:3.0.7 redis  

 

For the detailed process of the update, and the effect of the parameters, see the following figure:

 

 

Now, the mirror of redis 3.0.7 version has been used.

 

By default, the scheduler uses the following rolling update strategy:

 

  1. stop the first task
  2. Update stopped task s
  3. Start the updated task
  4. If the updated task is successful, return to RUNNING, then stop, wait for a delay period specified by the --update-delay parameter, and then update the next task.
  5. If at any time during the update process, just return to the FAILED state and suspend the update.

 

Notice! If an update fails, where can I see why it failed?

 

Just need docker service inspect , for example:

 

 

plus docker service ps

 

 

You can see the specific reason for the failure of the update.

 

If an update fails, restart the update, what to do?

 

docker service update redis
 

To view the rolling upgrade process, execute the following command:

 

docker service ps redis

 

 

 

Shown here is the state after the final update is complete.

 

In addition, if there are many instances of the service, during the update process, if you execute docker service ps to view the status of the service, you will see that some tasks are running the 3.0.6 version of the image, and some task instances are running 3.0. 7 mirrors.

 

At this point, the rolling upgrade of the service is over. If there is anything you don't understand, please leave a message. Anyway, it may not answer you.

Posted by afterburner on Thu, 01 Sep 2022 02:20:21 +0930