Authors: Cheng Dehao, Fluent Member, KubeSphere Member
Introduction to Fluent Operator
With the rapid development of cloud native technology and the continuous iteration of technology, higher requirements are put forward for log collection, processing and forwarding. The log scheme under the cloud native architecture is very different from the log architecture design based on physical machine or virtual machine scenario. As a graduation project of CNCF, Fluent Bit is undoubtedly one of the preferred solutions to solve the logging problem in cloud environment. The deployment cost of bitnet and the deployment cost of kunette users have increased to a certain extent.
On January 21, 2019, KubeSphere community developed the Fluent Bit operator to meet the needs of managing Fluent Bit in a cloud native way, and released v0.0 on February 17, 2020 Version 1.0. Since then, the product has been iterative, and the Fluent Bit operator was officially donated to the Fluent community on August 4, 2021.
Fluent Bit operator reduces the threshold of using Fluent Bit and can process log information efficiently and quickly. However, the ability of Fluent Bit to process logs is slightly weak. We have not integrated log processing tools, such as fluent D, which has more plug-ins to use. Based on the above requirements, Fluent Bit operator integrates fluent D, which aims to integrate fluent d into an optional log aggregation and forwarding layer and rename it Fluent Operator (GitHub address: https://github.com/fluent/fluent-operator) . On March 25, 2022, Fluent Operator released v1 Version 0.0, and will continue to iterate the Fluent Operator. V1.0 is expected to be released in the second quarter of 2022 Version 1.0 adds more functions and highlights.
Using Fluent Operator, you can deploy, configure and uninstall Fluent Bit and fluent D flexibly and easily. At the same time, the community also provides a large number of plug-ins that support fluent D and Fluent Bit. Users can customize the configuration according to the actual situation. The official documents provide detailed examples, which are very easy to use and greatly reduce the threshold of using Fluent Bit and fluent D.
Each stage of the Log Pipeline
The Fluent Operator can deploy Fluent Bit or fluent D separately, and will not force the use of Fluent Bit or fluent D. at the same time, it also supports the use of fluent d to receive the log stream forwarded by Fluent Bit for multi tenant log isolation, which greatly increases the flexibility and diversity of deployment. In order to have a more comprehensive understanding of Fluent Operator, the following figure takes the complete log pipeline as an example and divides the pipeline into three parts: collection and forwarding, filtering and output.
Collection and forwarding
Both Fluent Bit and fluent D can collect logs.
When deployed separately, you can meet the needs of log collection through the int put plug-in of Fluent Bit or the forward and http plug-ins of fluent D. When the two are combined, fluent D can use forward to accept the log stream forwarding of Fluent Bit.
In terms of performance, compared with fluent D, Fluent Bit is lighter and consumes less memory (about 650KB), so Fluent Bit is mainly responsible for collecting and forwarding logs. Collect and forward logs on each node through Fluent Bit installed in the form of daemon set.
filter
The data collected by the log is often too messy and redundant, which requires the log processing middleware to provide the ability to filter and process the log information. Both Fluent Bit and fluent D support filter plug-in. Users can integrate and customize log data according to their own needs.
output
The Fluent Bit output or fluent D output plug-in outputs the processed log information to multiple destinations, which can be third-party components such as Kafka and Elasticsearch.
Introduction to CRD
Fluent Operator defines two groups for Fluent Bit and fluent D: Fluent Bit fluent. IO and fluent D fluent. io.
fluentbit.fluent.io
fluentbit.fluent.io# the following six CRDs are included under the group:
- Fluent Bit CRD defines the attributes of Fluent Bit, such as image version, stain, affinity and other parameters.
- Clusterfluent bitconfig CRD defines the configuration file of Fluent Bit.
- ClusterInput CRD defines the input plug-in of Fluent Bit, that is, the input plug-in. Through this plug-in, users can customize which logs to collect.
- ClusterFilter CRD defines the filter plug-in of Fluent Bit, which is mainly responsible for filtering and processing the information collected by Fluent Bit.
- ClusterParser CRD defines the parser plug-in of Fluent Bit, which is mainly responsible for parsing log information and can parse log information into other formats.
- ClusterOutput CRD defines the output plug-in of Fluent Bit, which is mainly responsible for forwarding the processed log information to the destination.
fluentd.fluent.io
fluentd.fluent.io# the following seven CRDs are included under the group:
- Fluentd CRD defines the attributes of fluentd, such as image version, stain, affinity and other parameters.
- Clusterfluent dconfig CRD defines the configuration file of fluent cluster level.
- Fluent dconfig CRD defines the configuration file of fluent's namespace scope.
- ClusterFilter CRD defines the filter plug-in in the scope of fluent cluster, which is mainly responsible for filtering and processing the information collected by fluent. If Fluent Bit is installed, log information can be further processed.
- Filter CRD this CRD defines the filter plug-in of fluent namespace, which is mainly responsible for filtering and processing the information collected by fluent. If Fluent Bit is installed, log information can be further processed.
- ClusterOutput CRD this CRD defines the cluster wide output plug-in of fluent, which is mainly responsible for forwarding the processed log information to the destination.
- Output CRD this CRD defines the output plug-in in the namespace scope of fluent, which is mainly responsible for forwarding the processed log information to the destination.
Introduction to choreography principle (instance + mounted secret + abstraction ability of CRD)
Although Fluent Bit and fluent d have the ability to collect, process (parse and filter) and output logs, they have different advantages. Fluent Bit is lighter and more efficient than fluent D, and fluent D has more plug-ins.
In order to balance these advantages, Fluent Operator allows users to flexibly use Fluent Bit and fluent D in a variety of ways:
- Fluent Bit only mode: if you only need to collect logs and send them to the final destination after simple processing, you only need Fluent Bit.
- Fluent only mode: if you need to receive logs through the network in the form of HTTP or Syslog, and then process and send the logs to the final destination, you only need fluent.
- Fluent Bit + fluent D mode: if you need to do some advanced processing on the collected logs or send them to more sink s, you can use Fluent Bit and fluent D in combination.
Fluent Operator allows you to configure the log processing pipeline in the above three modes as required. Fluent D and Fluent Bit have rich plug-ins to meet various customization needs of users. Since the configuration and mounting methods of fluent D and Fluent Bit are similar, the mounting method of Fluent Bit configuration file is briefly introduced.
In the Fluent Bit CRD, each clusterinput, clusterparser, clusterfilter and clusteroutput represents a Fluent Bit configuration part, which is selected by the clusterfluent bitconfig tag selector. The Fluent Operator monitors these objects, builds the final configuration, and finally creates a Secret to store the configuration installed in the Fluent Bit daemon set. The whole workflow is as follows:
Because the Fluent Bit itself does not reload the interface (please refer to this for details) Known problems ), in order to enable Fluent Bit to obtain and use the latest configuration when the configuration of Fluent Bit is changed, a wrapper named Fluent Bit watcher is added to restart the Fluent Bit process immediately when the configuration change of Fluent Bit is detected. In this way, the new configuration can be reloaded without restarting the Fluent Bit pod.
In order to make user configuration more convenient, we extract the parameters of application and configuration based on the powerful abstraction ability of CRD. Users can configure Fluent Bit and fluent D through defined CRD. The Fluent Operator monitors the changes of these objects to change the status and configuration of Fluent Bit and fluent D. Especially for the definition of plug-ins, in order to make the user transition more smoothly, we basically keep the naming consistent with the original fields of Fluent Bit to reduce the threshold of use.
How to realize multi tenant log isolation
Fluent Bit can efficiently collect logs, but if it needs to carry out complex processing of log information, Fluent Bit is a little weak, while fluent D can complete advanced processing of log information with the help of its rich plug-ins. Fluent operator abstracts various plug-ins of fluent d so that the log information can be processed to meet the user-defined requirements.
As can be seen from the above definition of CRD, we divide the config uration of fluent D and the CRD of plug-in into cluster level and namespace level CRDs. By defining CRD as two ranges and using the label router plug-in of fluent, the effect of multi tenant isolation can be achieved.
We added the watchNamespace field in clusterfluent config. Users can choose which namespaces to listen to according to their needs. If it is empty, it means to monitor all namespaces. The nameapce level fluent config can only listen to the CR and global level configuration in the namespace in which it is located. Therefore, the log at the namespace level can be output to both the output in the namespace and the output at the clsuter level, so as to achieve the purpose of multi tenant isolation.
Fluent Operator vs logging-operator
difference
- Both can automatically deploy Fluent Bit and fluent D. The logging operator needs to deploy Fluent Bit and Fluent Bit at the same time, while the Fluent Operator supports pluggable deployment of Fluent Bit and fluent D, which are not strongly coupled. Users can choose to deploy Fluent Bit or Fluent Bit according to their own needs, which is more flexible.
- In the logging operator, the logs collected by Fluent Bit must pass through fluent d before they can be output to the final destination. Moreover, if the amount of data is too large, fluent D has a single point of failure. The Fluent Bit in the Fluent Operator can directly send the log information to the destination, so as to avoid the hidden danger of single point of failure.
- Logging operator defines four CRDs: logging, outputs, flows, clusteroutputs and clusterflows, while Fluent Operator defines 13 CRDs. Compared with the logging operator, the Fluent Operator has more diverse CRD definitions. Users can configure fluent D and Fluent Bit more flexibly according to their needs. At the same time, when defining CRD, select the name similar to the configuration of fluent D and Fluent Bit, and strive to make the name clearer to meet the original component definition.
- Both use the label router plug-in of fluent for reference to realize multi tenant log isolation.
Outlook:
- Support HPA automatic expansion and contraction;
- Improve Helm Chart, such as collecting metrics information;
- ...
would rather
With the help of Fluent Operator, we can carry out complex processing of logs. Here, we can fluent-operator-walkthrough The examples of outputting logs to elasticsearch and Kafka introduce the actual functions of Fluent Operator. To gain some hands-on experience with Fluent Operator, you need a Kind cluster. At the same time, you also need to set up a Kafka cluster and an elasticsearch cluster in this type of cluster.
# Create a kind cluster and name it fluent ./create-kind-cluster.sh # Create a Kafka cluster under Kafka namespace ./deploy-kafka.sh # Create an elastic search cluster under the elastic namespace ./deploy-es.sh
The Fluent Operator controls the lifecycle of Fluent Bit and fluent D. You can start the Fluent Operator in the fluent namespace using the following script:
./deploy-fluent-operator.sh
Both Fluent Bit and fluent d have been defined as CRD s in Fluent Operator. You can create Fluent Bit DaemonSet or fluent statefulset by declaring the CR of Fluent Bit or fluent D.
Fluent Bit Only mode
Fluent Bit Only will only enable lightweight Fluent Bit to collect, process and forward logs.
Use Fluent Bit to collect kubelet's logs and output them to Elasticsearch
cat <<EOF | kubectl apply -f - apiVersion: fluentbit.fluent.io/v1alpha2 kind: FluentBit metadata: name: fluent-bit namespace: fluent labels: app.kubernetes.io/name: fluent-bit spec: image: kubesphere/fluent-bit:v1.8.11 positionDB: hostPath: path: /var/lib/fluent-bit/ resources: requests: cpu: 10m memory: 25Mi limits: cpu: 500m memory: 200Mi fluentBitConfigName: fluent-bit-only-config tolerations: - operator: Exists --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterFluentBitConfig metadata: name: fluent-bit-only-config labels: app.kubernetes.io/name: fluent-bit spec: service: parsersFile: parsers.conf inputSelector: matchLabels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "fluentbit-only" filterSelector: matchLabels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "fluentbit-only" outputSelector: matchLabels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "fluentbit-only" --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterInput metadata: name: kubelet labels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "fluentbit-only" spec: systemd: tag: service.kubelet path: /var/log/journal db: /fluent-bit/tail/kubelet.db dbSync: Normal systemdFilter: - _SYSTEMD_UNIT=kubelet.service --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterFilter metadata: name: systemd labels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "fluentbit-only" spec: match: service.* filters: - lua: script: key: systemd.lua name: fluent-bit-lua call: add_time timeAsTable: true --- apiVersion: v1 data: systemd.lua: | function add_time(tag, timestamp, record) new_record = {} timeStr = os.date("!*t", timestamp["sec"]) t = string.format("%4d-%02d-%02dT%02d:%02d:%02d.%sZ", timeStr["year"], timeStr["month"], timeStr["day"], timeStr["hour"], timeStr["min"], timeStr["sec"], timestamp["nsec"]) kubernetes = {} kubernetes["pod_name"] = record["_HOSTNAME"] kubernetes["container_name"] = record["SYSLOG_IDENTIFIER"] kubernetes["namespace_name"] = "kube-system" new_record["time"] = t new_record["log"] = record["MESSAGE"] new_record["kubernetes"] = kubernetes return 1, timestamp, new_record end kind: ConfigMap metadata: labels: app.kubernetes.io/component: operator app.kubernetes.io/name: fluent-bit-lua name: fluent-bit-lua namespace: fluent --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterOutput metadata: name: es labels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "fluentbit-only" spec: matchRegex: (?:kube|service)\.(.*) es: host: elasticsearch-master.elastic.svc port: 9200 generateID: true logstashPrefix: fluent-log-fb-only logstashFormat: true timeKey: "@timestamp" EOF
Use Fluent Bit to collect the application log of kubernetes and output it to Kafka
cat <<EOF | kubectl apply -f - apiVersion: fluentbit.fluent.io/v1alpha2 kind: FluentBit metadata: name: fluent-bit namespace: fluent labels: app.kubernetes.io/name: fluent-bit spec: image: kubesphere/fluent-bit:v1.8.11 positionDB: hostPath: path: /var/lib/fluent-bit/ resources: requests: cpu: 10m memory: 25Mi limits: cpu: 500m memory: 200Mi fluentBitConfigName: fluent-bit-config tolerations: - operator: Exists --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterFluentBitConfig metadata: name: fluent-bit-config labels: app.kubernetes.io/name: fluent-bit spec: service: parsersFile: parsers.conf inputSelector: matchLabels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "k8s" filterSelector: matchLabels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "k8s" outputSelector: matchLabels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "k8s" --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterInput metadata: name: tail labels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "k8s" spec: tail: tag: kube.* path: /var/log/containers/*.log parser: docker refreshIntervalSeconds: 10 memBufLimit: 5MB skipLongLines: true db: /fluent-bit/tail/pos.db dbSync: Normal --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterFilter metadata: name: kubernetes labels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/mode: "k8s" spec: match: kube.* filters: - kubernetes: kubeURL: https://kubernetes.default.svc:443 kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token labels: false annotations: false - nest: operation: lift nestedUnder: kubernetes addPrefix: kubernetes_ - modify: rules: - remove: stream - remove: kubernetes_pod_id - remove: kubernetes_host - remove: kubernetes_container_hash - nest: operation: nest wildcard: - kubernetes_* nestUnder: kubernetes removePrefix: kubernetes_ --- apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterOutput metadata: name: kafka labels: fluentbit.fluent.io/enabled: "false" fluentbit.fluent.io/mode: "k8s" spec: matchRegex: (?:kube|service)\.(.*) kafka: brokers: my-cluster-kafka-bootstrap.kafka.svc:9091,my-cluster-kafka-bootstrap.kafka.svc:9092,my-cluster-kafka-bootstrap.kafka.svc:9093 topics: fluent-log EOF
Fluent bit + fluent D mode
With its rich plug-ins, Fluentd can act as a log aggregation layer to perform more advanced log processing. You can easily forward logs from Fluent Bit to fluent d using Fluent Operator.
Forward logs from Fluent Bit to fluent D
To forward logs from Fluent Bit to fluent D, you need to enable the forward plug-in of Fluent Bit, as shown below:
cat <<EOF | kubectl apply -f - apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterOutput metadata: name: fluentd labels: fluentbit.fluent.io/enabled: "true" fluentbit.fluent.io/component: logging spec: matchRegex: (?:kube|service)\.(.*) forward: host: fluentd.fluent.svc port: 24224 EOF
Deploy fluent
The fluent forward input plug-in will be enabled by default when deploying fluent, so you only need to deploy the following yaml to deploy fluent:
apiVersion: fluentd.fluent.io/v1alpha1 kind: Fluentd metadata: name: fluentd namespace: fluent labels: app.kubernetes.io/name: fluentd spec: globalInputs: - forward: bind: 0.0.0.0 port: 24224 replicas: 1 image: kubesphere/fluentd:v1.14.4 fluentdCfgSelector: matchLabels: config.fluentd.fluent.io/enabled: "true"
ClusterFluentdConfig: Fluentd cluster wide configuration
If you define clusterfluent dconfig, you can collect logs in any or all namespaces. We can select the namespace to collect logs through the watchedNamespaces field. The following configuration is to collect logs under Kube system and default namespace:
cat <<EOF | kubectl apply -f - apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterFluentdConfig metadata: name: cluster-fluentd-config labels: config.fluentd.fluent.io/enabled: "true" spec: watchedNamespaces: - kube-system - default clusterOutputSelector: matchLabels: output.fluentd.fluent.io/scope: "cluster" output.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterOutput metadata: name: cluster-fluentd-output-es labels: output.fluentd.fluent.io/scope: "cluster" output.fluentd.fluent.io/enabled: "true" spec: outputs: - elasticsearch: host: elasticsearch-master.elastic.svc port: 9200 logstashFormat: true logstashPrefix: fluent-log-cluster-fd EOF
Fluent dconfig: configuration of fluent namespace scope
If you define fluent dconfig, you can only send logs in the same namespace as the fluent dconfig to Output. In this way, you can isolate logs in different namespaces.
cat <<EOF | kubectl apply -f - apiVersion: fluentd.fluent.io/v1alpha1 kind: FluentdConfig metadata: name: namespace-fluentd-config namespace: fluent labels: config.fluentd.fluent.io/enabled: "true" spec: outputSelector: matchLabels: output.fluentd.fluent.io/scope: "namespace" output.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: Output metadata: name: namespace-fluentd-output-es namespace: fluent labels: output.fluentd.fluent.io/scope: "namespace" output.fluentd.fluent.io/enabled: "true" spec: outputs: - elasticsearch: host: elasticsearch-master.elastic.svc port: 9200 logstashFormat: true logstashPrefix: fluent-log-namespace-fd EOF
Route logs to different Kafka topic s according to the namespace
Similarly, you can use the filter plug-in of fluent to distribute logs to different topic s according to different namespace s. Here, we include the plug-in recordTransformer in the fluent kernel, which can add, delete, modify and check events.
cat <<EOF | kubectl apply -f - apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterFluentdConfig metadata: name: cluster-fluentd-config-kafka labels: config.fluentd.fluent.io/enabled: "true" spec: watchedNamespaces: - kube-system - default clusterFilterSelector: matchLabels: filter.fluentd.fluent.io/type: "k8s" filter.fluentd.fluent.io/enabled: "true" clusterOutputSelector: matchLabels: output.fluentd.fluent.io/type: "kafka" output.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterFilter metadata: name: cluster-fluentd-filter-k8s labels: filter.fluentd.fluent.io/type: "k8s" filter.fluentd.fluent.io/enabled: "true" spec: filters: - recordTransformer: enableRuby: true records: - key: kubernetes_ns value: ${record["kubernetes"]["namespace_name"]} --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterOutput metadata: name: cluster-fluentd-output-kafka labels: output.fluentd.fluent.io/type: "kafka" output.fluentd.fluent.io/enabled: "true" spec: outputs: - kafka: brokers: my-cluster-kafka-bootstrap.default.svc:9091,my-cluster-kafka-bootstrap.default.svc:9092,my-cluster-kafka-bootstrap.default.svc:9093 useEventTime: true topicKey: kubernetes_ns EOF
Use both cluster wide and namespace wide fluent dconfig
Of course, you can use both clusterfluent dconfig and fluent dconfig as follows. FluentdConfig will send the log under the fluent namespace to ClusterOutput, and ClusterFluentdConfig will also send the namespace under the watchedNamespaces field (i.e. Kube system and default namespaces) to ClusterOutput.
cat <<EOF | kubectl apply -f - apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterFluentdConfig metadata: name: cluster-fluentd-config-hybrid labels: config.fluentd.fluent.io/enabled: "true" spec: watchedNamespaces: - kube-system - default clusterOutputSelector: matchLabels: output.fluentd.fluent.io/scope: "hybrid" output.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: FluentdConfig metadata: name: namespace-fluentd-config-hybrid namespace: fluent labels: config.fluentd.fluent.io/enabled: "true" spec: clusterOutputSelector: matchLabels: output.fluentd.fluent.io/scope: "hybrid" output.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterOutput metadata: name: cluster-fluentd-output-es-hybrid labels: output.fluentd.fluent.io/scope: "hybrid" output.fluentd.fluent.io/enabled: "true" spec: outputs: - elasticsearch: host: elasticsearch-master.elastic.svc port: 9200 logstashFormat: true logstashPrefix: fluent-log-hybrid-fd EOF
In the multi tenant scenario, the cluster wide and namespace wide FluentdConfig are used at the same time
In the multi tenant scenario, we can use both cluster wide and namespace wide fluent dconfig to achieve the effect of log isolation.
cat <<EOF | kubectl apply -f - apiVersion: fluentd.fluent.io/v1alpha1 kind: FluentdConfig metadata: name: namespace-fluentd-config-user1 namespace: fluent labels: config.fluentd.fluent.io/enabled: "true" spec: outputSelector: matchLabels: output.fluentd.fluent.io/enabled: "true" output.fluentd.fluent.io/user: "user1" clusterOutputSelector: matchLabels: output.fluentd.fluent.io/enabled: "true" output.fluentd.fluent.io/user: "user1" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterFluentdConfig metadata: name: cluster-fluentd-config-cluster-only labels: config.fluentd.fluent.io/enabled: "true" spec: watchedNamespaces: - kube-system - kubesphere-system clusterOutputSelector: matchLabels: output.fluentd.fluent.io/enabled: "true" output.fluentd.fluent.io/scope: "cluster-only" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: Output metadata: name: namespace-fluentd-output-user1 namespace: fluent labels: output.fluentd.fluent.io/enabled: "true" output.fluentd.fluent.io/user: "user1" spec: outputs: - elasticsearch: host: elasticsearch-master.elastic.svc port: 9200 logstashFormat: true logstashPrefix: fluent-log-user1-fd --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterOutput metadata: name: cluster-fluentd-output-user1 labels: output.fluentd.fluent.io/enabled: "true" output.fluentd.fluent.io/user: "user1" spec: outputs: - elasticsearch: host: elasticsearch-master.elastic.svc port: 9200 logstashFormat: true logstashPrefix: fluent-log-cluster-user1-fd --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterOutput metadata: name: cluster-fluentd-output-cluster-only labels: output.fluentd.fluent.io/enabled: "true" output.fluentd.fluent.io/scope: "cluster-only" spec: outputs: - elasticsearch: host: elasticsearch-master.elastic.svc port: 9200 logstashFormat: true logstashPrefix: fluent-log-cluster-only-fd EOF
Use buffer for fluent output
You can add a buffer to cache the logs of the output plug-in.
cat <<EOF | kubectl apply -f - apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterFluentdConfig metadata: name: cluster-fluentd-config-buffer labels: config.fluentd.fluent.io/enabled: "true" spec: watchedNamespaces: - kube-system - default clusterFilterSelector: matchLabels: filter.fluentd.fluent.io/type: "buffer" filter.fluentd.fluent.io/enabled: "true" clusterOutputSelector: matchLabels: output.fluentd.fluent.io/type: "buffer" output.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterFilter metadata: name: cluster-fluentd-filter-buffer labels: filter.fluentd.fluent.io/type: "buffer" filter.fluentd.fluent.io/enabled: "true" spec: filters: - recordTransformer: enableRuby: true records: - key: kubernetes_ns value: ${record["kubernetes"]["namespace_name"]} --- apiVersion: fluentd.fluent.io/v1alpha1 kind: ClusterOutput metadata: name: cluster-fluentd-output-buffer labels: output.fluentd.fluent.io/type: "buffer" output.fluentd.fluent.io/enabled: "true" spec: outputs: - stdout: {} buffer: type: file path: /buffers/stdout.log - elasticsearch: host: elasticsearch-master.elastic.svc port: 9200 logstashFormat: true logstashPrefix: fluent-log-buffer-fd buffer: type: file path: /buffers/es.log EOF
Fluent only mode
You can also enable the fluent only mode, which will only deploy fluent statefulset.
Use fluent to receive logs from HTTP and output them to standard output
If you want to open the fluent plug-in separately, you can receive logs through HTTP.
cat <<EOF | kubectl apply -f - apiVersion: fluentd.fluent.io/v1alpha1 kind: Fluentd metadata: name: fluentd-http namespace: fluent labels: app.kubernetes.io/name: fluentd spec: globalInputs: - http: bind: 0.0.0.0 port: 9880 replicas: 1 image: kubesphere/fluentd:v1.14.4 fluentdCfgSelector: matchLabels: config.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: FluentdConfig metadata: name: fluentd-only-config namespace: fluent labels: config.fluentd.fluent.io/enabled: "true" spec: filterSelector: matchLabels: filter.fluentd.fluent.io/mode: "fluentd-only" filter.fluentd.fluent.io/enabled: "true" outputSelector: matchLabels: output.fluentd.fluent.io/mode: "true" output.fluentd.fluent.io/enabled: "true" --- apiVersion: fluentd.fluent.io/v1alpha1 kind: Filter metadata: name: fluentd-only-filter namespace: fluent labels: filter.fluentd.fluent.io/mode: "fluentd-only" filter.fluentd.fluent.io/enabled: "true" spec: filters: - stdout: {} --- apiVersion: fluentd.fluent.io/v1alpha1 kind: Output metadata: name: fluentd-only-stdout namespace: fluent labels: output.fluentd.fluent.io/enabled: "true" output.fluentd.fluent.io/enabled: "true" spec: outputs: - stdout: {} EOF
This article is composed of blog one article multi posting platform OpenWrite release!