Use MinIO Gateway and MinIO Client to adapt Tencent cloud COS

1. General

1.1 INTRODUCTION

Starting from the general AWS S3 protocol and the scenario of using COS in S3 compatible third-party applications, this paper introduces the steps of configuring Tencent cloud object storage cos based on MinIO object storage, including the preparation of compilation environment and the configuration of MinIO Client and MinIO Gateway.

1.2 interpretation of terms

Cos: Cloud Object Storage (COS) is a distributed Storage service launched by Tencent Cloud, which has no directory hierarchy, no data format restrictions, can accommodate massive data and supports HTTP/HTTPS protocol access. The Storage bucket space of Tencent Cloud # cos # has no upper capacity limit and does not need partition management. It is suitable for # CDN # data distribution, data Vientiane processing or data lake for big data calculation and analysis. Cos ， provides web side management interface, SDK and API of various mainstream development languages, command line and graphical tools, and is compatible with ， API ， interface of ， S3 ， to facilitate users to directly use community tools and plug-ins.

AWS S3: Amazon Simple Storage Service (Amazon S3) is an object storage service that provides industry-leading scalability, data availability, security and performance. Customers of all sizes and industries can store and protect any amount of data for almost any use case, such as data lakes, cloud native applications and mobile applications. With cost-effective storage classes and easy-to-use management capabilities, you can optimize costs, organize data, and configure finely tuned access controls to meet specific business, organizational, and compliance requirements.

MinIO object storage: MinIO is a high performance, distributed object storage system It is a software product that can run 100% on standard hardware. That is, low-cost machines such as X86 can also run MinIO well. The difference between MinIO and traditional storage and other object storage is that it designs the software architecture for the private cloud standard with higher performance requirements from the beginning. Because MinIO was designed only for object storage from the beginning. Therefore, it adopts a more easy-to-use way to design. It can realize all the functions required by object storage and has stronger performance. It will not compromise for more business functions and lose the ease of use and efficiency of MinIO. The benefit of this result is that it can more simply implement the local elastic and scalable native object storage service. MinIO excels in traditional object storage use cases such as secondary storage, disaster recovery and archiving. At the same time, it is also unique in the storage technology of machine learning, big data, private cloud, hybrid cloud and so on. Of course, it does not rule out the support of data analysis, high-performance application load and native cloud.

MinIO Client: MinIO Client (MC for short) provides an alternative to UNIX commands (such as ls, cat, cp, mirror, diff, find, etc.). It supports file system level operations and is compatible with AWS S3 protocol cloud storage services (including AWS Signature v2 and v4)

MinIO Gateway: MinIO S3 Gateway adds MinIO functions (such as MinIO console and disk cache) to AWS S3 or cloud storage services compatible with AWS S3 protocol. To avoid increasing the complexity of adapting a variety of cloud storage products in SDK development, MinIO Gateway can be used as a layer of agent, and the SDK only needs to support the access mode of MinIO; At the same time, it also supports various account management methods, such as keyloak, LDAP, etc.

2.MinIO architecture

2.1 typical business architecture diagram of Minio

2.2 architecture diagram of Minio gateway

3. Adapt Tencent cloud COS

3.1 why is it necessary to compile separately to adapt Tencent cloud COS

MinIO Gateway and MinIO Client will randomly generate a Bucket name when they are started, and then initiate GetBucketLocation Request to check whether the S3 service is available. If the returned status code is 404, the verification is successful.

However, the style name of the Bucket of Tencent cloud COS is < bucketname > - < appid >. If it does not conform to the format, it will respond to 400, resulting in the failure of MinIO Gateway or MinIO Client verification.

The packet capture information is as follows

GET /probe-bucket-sign-u5x32il6o626/?location= HTTP/1.1
Host: cos.ap-beijing.myqcloud.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.23
Authorization: AWS4-HMAC-SHA256 Credential=AKIDe5e18zmPnxsZ1EDwRmskNkskMipgJhro/20220414/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a22e2d2c171d4f649f7439fc8e53acae916065ecf7c8c55b804318717cc06dc0
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220414T185329Z

HTTP/1.1 400 Bad Request
Content-Type: application/xml
Content-Length: 433
Connection: keep-alive
Date: Thu, 14 Apr 2022 18:53:30 GMT
Server: tencent-cos
x-amz-request-id: NjI1ODZkYTlfOWNiNTE0MGJfYzFmXzFjNmI0MDE=
x-amz-trace-id: OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc1NGE1MWY0MzY2NTg1MzM1OTY3MDliYzY2YTQ0ZThhMDM2NjEzNmJhN2E5MjY5MDAxNWE1NzdhNGVhNzcyMGVm

<?xml version='1.0' encoding='utf-8' ?>
<Error>
	<Code>InvalidURI</Code>
	<Message>Could not parse the specified URI.</Message>
	<Resource>cos.ap-beijing.myqcloud.com/probe-bucket-sign-u5x32il6o626</Resource>
	<RequestId>NjI1ODZkYTlfOWNiNTE0MGJfYzFmXzFjNmI0MDE=</RequestId>
	<TraceId>OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc1NGE1MWY0MzY2NTg1MzM1OTY3MDliYzY2YTQ0ZThhMDM2NjEzNmJhN2E5MjY5MDAxNWE1NzdhNGVhNzcyMGVm</TraceId>
</Error>

3.2 how does Minio gateway adapt to Tencent cloud COS

First, execute the following command to download the MinIO source code

URL https://github.com/minio/minio
git clone https://github.com/minio/minio.git

At / CMD / gateway / S3 / gateway-s3 Add "- 123456" at the end of line 139 of go file

L124 // randString generates random names and prepends them with a known prefix.
L125 func randString(n int, src rand.Source, prefix string)
L139 return prefix + string(b[0:30-len(prefix)]) + "-123456"

It can be used after compiling

Compiled MinIO Gateway For Cos UNIX binary executable

https://cos.iclay.cn/Software/Linux/minio/minio-for-cos

3.3MinIO Client how to adapt Tencent cloud COS

First, execute the following command to download the source code of MinIO Client

URL https://github.com/minio/mc/
git clone https://github.com/minio/mc.git

In CMD / utils Add "- 123456" at the end of line 105 of the go file

L90 // randString generates random names and prepends them with a known prefix.
L91 func randString(n int, src rand.Source, prefix string)
L105 return prefix + string(b[0:30-len(prefix)]) + "-123456"