TorchServe deploys yolov5 project (Docker article)
I. Introduction
Nothing to say, otherwise you won't be interested in this blog
2. Packing mar
1. Download the project
#Download the branch mini version (it is recommended to start with the mini version first, and then use the completed version to verify) git clone -b mini https://github.com/lbz2020/torchserve_yolov5.git #Download full version git clone https://github.com/lbz2020/torchserve_yolov5.git
2. Enter the project main directory torchserve_yolov5 (the terminal command window, I am operating under Mac OS, I did not take care of Windows users)
Note: If there is no special description or designation, the relative path of the full text command is performed in the main directory of the project
3. Install dependencies
#upgrade pip pip install --upgrade pip #Installation dependencies (use domestic mirror acceleration) pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
4. Enter the subdirectory yolov5 and convert the .pt file into a .torchscript file
#Open subdirectory yolov5 cd yolov5 #Export .torchscript file from .pt file python export.py --weights yolov5s.pt --include torchscript
5. Go back to the main project directory
#Return to the previous directory cd ..
6. Packaged into .mar file
#Package all files into .mar file torch-model-archiver --model-name demo --version 1.0 --serialized-file yolov5/yolov5s.torchscript --handler yolov5_model_handler.py --extra-files index_to_name.json
torch-model-archiver parameter explanation
-
--model-name : model name demo
-
--version: Model version number 1.0
-
--serialized-file: .torchscript file generated by .pt yolov5/yolov5s.torchscript
-
--handler The processor for the yolov5 model, including preprocessing, inference, postprocessing, etc. yolov5_model_handler.py
-
--extra-files Extra files packaged together, here contains the tag index mapping file index_to_name.json
After executing the above command, a demo.mar file will be generated in the main directory of the project
7. Create a folder model-store in the main project folder, and then put the demo.mar file in the folder model-store. The purpose is to do a folder mapping when the container is started later.
#Note: The following are Linux commands!!! #Create a folder model-store mkdir model-store #Move the demo.mar file to the folder model-store mv demo.mar ./model-store
3. docker image
It is not recommended to directly pull the docker image provided by the official
#Pull the official torchserve image (don't care) docker pull pytorch/torchserve
Note: Why not just use the official image provided? Because only the most basic python dependencies are provided in the official image. We generally use other dependencies, such as opencv, when writing handler files. These dependencies are not provided by the official docker image. Later, I will make a public image and put it on dockerhub for everyone to pull.
1. Suppose we all downloaded and installed docker
2. Make a docker image of TorchServe
The next thing we need to do is to add some python dependencies of yolov5 to the official Dockerfile
RUN python -m pip install matplotlib>=3.2.2 numpy>=1.18.5 opencv-python>=4.1.1 Pillow>=7.1.2 PyYAML>=5.3.1 requests>=2.23.0 scipy>=1.4.1 torch>=1.7.0 torchvision>=0.8.1 tqdm>=4.64.0 tensorboard>=2.4.1 pandas>=1.1.4 seaborn>=0.11.0 ipython psutil thop>=0.1.1 -i https://pypi.tuna.tsinghua.edu.cn/simple/
a. Modify the Dockerfile
Find the Dockerfile file: in the main directory /serve/docker folder
Then open the Dockerfile, locate the 76th line, and add the following command
RUN python -m pip install matplotlib>=3.2.2 numpy>=1.18.5 opencv-python>=4.1.1 Pillow>=7.1.2 PyYAML>=5.3.1 requests>=2.23.0 scipy>=1.4.1 torch>=1.7.0 torchvision>=0.8.1 tqdm>=4.64.0 tensorboard>=2.4.1 pandas>=1.1.4 seaborn>=0.11.0 ipython psutil thop>=0.1.1 -i https://pypi.tuna.tsinghua.edu.cn/simple/
b. Execute the command in the terminal
#Open directory path, home directory /serve/docker #It must be located in the docker directory, otherwise there will be many unknown problems cd serve/docker #Create an image (Linux) ./build_image.sh
windows has a lot of bug s, and it takes a long time to create an image. It took about an hour to create it, and then I didn’t try it.
two thousands years later ------------------------->>>>
On the docker panel, you can see the successfully created Docker image pytorch/torchserve:latest-cpu
end --------------------------------<<< !
4. Deployment
The docker command needs to be ** million dots **Hahahahahahahahahaha
1. Start the docker image
#Start command format (dev environment, will not run in the background) docker run --rm -it -p host host port:container internal port number --name container name -v host host folder:Folder inside the container Image name:Image tag number #Mirror made locally docker run --rm -it -p 8080:8080 -p 8081:8081 --name object_detector -v model-store folder path:/home/model-server/model-store pytorch/torchserve:latest-cpu #Mirror pull ed from my remote docker run --rm -it -p 8080:8080 -p 8081:8081 --name object_detector -v model-store folder path:/home/model-server/model-store pytorch/torchserve:v1.0 #Remember to change it to your own path
model-store folder path: the model-store folder mentioned in step 2 - 7
The internal folder path of the container /home/model-server/model-store This path is fixed and created when the image is built
so, is it ok to start the container, Mo De
Also register the model, otherwise there is nothing in the container. We operate through restapi.
2. Query the registered model
#Query registered models curl "http://localhost:8081/models"
3. Register the model and assign resources to the model
#Register the model and assign resources to the model curl -v -X POST "http://localhost:8081/models?initial_workers=1&synchronous=false&url=demo.mar"
4. View the basic information of the registered model
#View basic information of registered models curl "http://localhost:8081/models/demo"
5. Reasoning (it is time to witness miracles)
curl http://localhost:8080/predictions/demo -T docs/zidane.jpg
6. Log off the model (don't care)
curl -X DELETE http://localhost:8081/models/model name
And then, congratulations. I put the code and other stuff in it GitHub superior. Each of the above steps is written down after one step is completed by yourself, and then the next step is performed. So the project is fine. . . . . .
I will continue to update the documentation, there are many details that seem to have not been mentioned. There are also pictures of some operations that may be clearer after they are organized into the article. --20221012 init
V. Summary
There is nothing to sum up, a miracle in one night. sometimes watching official documentation very tired. . . . See more official documentation. If you have any questions, you can leave me a message
The difficulty of TorchServe lies in how to write handle.py. yolov5 itself is an engineering project, so a lot of engineering processing is written in handle. In fact, my project is still not perfect. . .
Six, issue
1. Is there a pit in the official image?
The official can not provide a model corresponding to the environment for each model, and each model has different dependencies.
2. Why choose TorchServe deployment? Instead of choosing TF Serving deployment, or choose Flask
The root cause is that he is too naive. . .
The direct reason Flask framework deploys AI model exists memory leak problem , causing another AI project to run very unstable in the production environment, and the Flask framework is easy to use, but it breaks down in scenarios with high concurrency.
I don't know if TF Serving is useful or not, but I'm useless anyway. I'm just a little chicken!
3. If you think of any problems, I will update them again. You can also leave a message to me.