TorchServe deploys yolov5 project (Docker article)

I. Introduction

​ Nothing to say, otherwise you won't be interested in this blog

2. Packing mar

1. Download the project

#Download the branch mini version (it is recommended to start with the mini version first, and then use the completed version to verify)
git clone -b mini https://github.com/lbz2020/torchserve_yolov5.git
#Download full version
git clone https://github.com/lbz2020/torchserve_yolov5.git

2. Enter the project main directory torchserve_yolov5 (the terminal command window, I am operating under Mac OS, I did not take care of Windows users)

​ Note: If there is no special description or designation, the relative path of the full text command is performed in the main directory of the project

3. Install dependencies

#upgrade pip
pip install --upgrade pip
#Installation dependencies (use domestic mirror acceleration)
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

4. Enter the subdirectory yolov5 and convert the .pt file into a .torchscript file

#Open subdirectory yolov5
cd yolov5
#Export .torchscript file from .pt file
python export.py --weights yolov5s.pt --include torchscript

5. Go back to the main project directory

#Return to the previous directory
cd ..

6. Packaged into .mar file

#Package all files into .mar file
torch-model-archiver --model-name demo --version 1.0 --serialized-file yolov5/yolov5s.torchscript --handler yolov5_model_handler.py --extra-files index_to_name.json

torch-model-archiver parameter explanation

• --model-name : model name demo

• --version: Model version number 1.0

• --serialized-file: .torchscript file generated by .pt yolov5/yolov5s.torchscript

• --handler The processor for the yolov5 model, including preprocessing, inference, postprocessing, etc. yolov5_model_handler.py

• --extra-files Extra files packaged together, here contains the tag index mapping file index_to_name.json

  After executing the above command, a demo.mar file will be generated in the main directory of the project

7. Create a folder model-store in the main project folder, and then put the demo.mar file in the folder model-store. The purpose is to do a folder mapping when the container is started later.

#Note: The following are Linux commands!!!
#Create a folder model-store
mkdir model-store
#Move the demo.mar file to the folder model-store
mv demo.mar ./model-store

3. docker image

It is not recommended to directly pull the docker image provided by the official

#Pull the official torchserve image (don't care)
docker pull pytorch/torchserve

Note: Why not just use the official image provided? Because only the most basic python dependencies are provided in the official image. We generally use other dependencies, such as opencv, when writing handler files. These dependencies are not provided by the official docker image. Later, I will make a public image and put it on dockerhub for everyone to pull.

1. Suppose we all downloaded and installed docker

2. Make a docker image of TorchServe

The next thing we need to do is to add some python dependencies of yolov5 to the official Dockerfile

RUN python -m pip install matplotlib>=3.2.2 numpy>=1.18.5 opencv-python>=4.1.1 Pillow>=7.1.2 PyYAML>=5.3.1 requests>=2.23.0 scipy>=1.4.1 torch>=1.7.0 torchvision>=0.8.1 tqdm>=4.64.0 tensorboard>=2.4.1 pandas>=1.1.4 seaborn>=0.11.0 ipython psutil thop>=0.1.1 -i https://pypi.tuna.tsinghua.edu.cn/simple/

​ a. Modify the Dockerfile

​ Find the Dockerfile file: in the main directory /serve/docker folder

​ Then open the Dockerfile, locate the 76th line, and add the following command

RUN python -m pip install matplotlib>=3.2.2 numpy>=1.18.5 opencv-python>=4.1.1 Pillow>=7.1.2 PyYAML>=5.3.1 requests>=2.23.0 scipy>=1.4.1 torch>=1.7.0 torchvision>=0.8.1 tqdm>=4.64.0 tensorboard>=2.4.1 pandas>=1.1.4 seaborn>=0.11.0 ipython psutil thop>=0.1.1 -i https://pypi.tuna.tsinghua.edu.cn/simple/

​ b. Execute the command in the terminal

#Open directory path, home directory /serve/docker
#It must be located in the docker directory, otherwise there will be many unknown problems
cd serve/docker
#Create an image (Linux)
./build_image.sh

​ windows has a lot of bug s, and it takes a long time to create an image. It took about an hour to create it, and then I didn’t try it.

two thousands years later ------------------------->>>>

On the docker panel, you can see the successfully created Docker image pytorch/torchserve:latest-cpu

end --------------------------------<<< !

4. Deployment

​ The docker command needs to be ** million dots **Hahahahahahahahahaha

1. Start the docker image

#Start command format (dev environment, will not run in the background)
docker run --rm -it -p host host port:container internal port number --name container name -v host host folder:Folder inside the container Image name:Image tag number

#Mirror made locally
docker run --rm -it -p 8080:8080 -p 8081:8081 --name object_detector -v model-store folder path:/home/model-server/model-store pytorch/torchserve:latest-cpu
#Mirror pull ed from my remote
docker run --rm -it -p 8080:8080 -p 8081:8081 --name object_detector -v model-store folder path:/home/model-server/model-store pytorch/torchserve:v1.0

#Remember to change it to your own path

​ model-store folder path: the model-store folder mentioned in step 2 - 7

​ The internal folder path of the container /home/model-server/model-store This path is fixed and created when the image is built

so, is it ok to start the container, Mo De

Also register the model, otherwise there is nothing in the container. We operate through restapi.

2. Query the registered model

#Query registered models
curl "http://localhost:8081/models"

3. Register the model and assign resources to the model

#Register the model and assign resources to the model
curl -v -X POST "http://localhost:8081/models?initial_workers=1&synchronous=false&url=demo.mar"

4. View the basic information of the registered model

#View basic information of registered models
curl "http://localhost:8081/models/demo"

5. Reasoning (it is time to witness miracles)

curl http://localhost:8080/predictions/demo -T docs/zidane.jpg

6. Log off the model (don't care)

curl -X DELETE http://localhost:8081/models/model name

And then, congratulations. I put the code and other stuff in it GitHub superior. Each of the above steps is written down after one step is completed by yourself, and then the next step is performed. So the project is fine. . . . . .

I will continue to update the documentation, there are many details that seem to have not been mentioned. There are also pictures of some operations that may be clearer after they are organized into the article. --20221012 init

V. Summary

​ There is nothing to sum up, a miracle in one night. sometimes watching official documentation very tired. . . . See more official documentation. If you have any questions, you can leave me a message

​

The difficulty of TorchServe lies in how to write handle.py. yolov5 itself is an engineering project, so a lot of engineering processing is written in handle. In fact, my project is still not perfect. . .

Six, issue

1. Is there a pit in the official image?

The official can not provide a model corresponding to the environment for each model, and each model has different dependencies.

2. Why choose TorchServe deployment? Instead of choosing TF Serving deployment, or choose Flask

The root cause is that he is too naive. . .

The direct reason Flask framework deploys AI model exists memory leak problem , causing another AI project to run very unstable in the production environment, and the Flask framework is easy to use, but it breaks down in scenarios with high concurrency.

I don't know if TF Serving is useful or not, but I'm useless anyway. I'm just a little chicken!

3. If you think of any problems, I will update them again. You can also leave a message to me.