在K8S上搭建filebeat+logstash+kafka采集多节点应用日志
2023-09-03 20:07:08

用途介绍

由于在k8s中的应用一般都会启动多个pod，导致在查询问题的时候需要挨个到每个pod里面查询日志。为了方便查询问题，需要将每个pod中的日志统一采集后集中存放在一个日志文件。因此在这里引入filebeat用于采集日志，logstash用于收集日志、筛选日志、处理日志，（可选）kafka将日志重新生产消费至其它地方。

准备工作

需要准备以下镜像，从docker官方仓库中获取：

  • filebeat:6.8.8：与ES版本相同，采集文件中的日志。
  • logstash:6.8.8：与ES版本相同，收集日志、解析日志。
  • kafka与ElasticSearch：使用现成已有的组件。

K8S配置文件

filebeat的yaml配置

注意：这里需要挂载到应用的yaml中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
kind: Deployment
apiVersion: apps/v1
metadata:
  name: app
  namespace: default
  labels:
    k8s-app: app
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: app
  template:
    metadata:
      name: app
      creationTimestamp: null
      labels:
        k8s-app: app
    spec:
      volumes:
        - name: storage-logs
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-app-configmap
            items:
              - key: filebeat-config
                path: filebeat.yml
            defaultMode: 420
      containers:
        - name: app
          image: 'app:0.1'
          command:
            - /bin/bash
            - '-c'
            - sh bin/start.sh
          args:
            - while true; do sleep 30; done;
          ports:
            - containerPort: 7101
              protocol: TCP
          resources:
            limits:
              cpu: '2'
              memory: 6Gi
            requests:
              cpu: '2'
              memory: 4Gi
          volumeMounts:
            - name: storage-logs
              mountPath: /app/logs
          imagePullPolicy: IfNotPresent
        - name: app-log
          image: 'filebeat:6.8.8'
          env:
            - name: TZ
              value: Asia/Shanghai
          resources:
            limits:
              cpu: 300m
              memory: 300Mi
            requests:
              cpu: 300m
              memory: 300Mi
          volumeMounts:
            - name: filebeat-config
              mountPath: /usr/share/filebeat/filebeat.yml
              subPath: filebeat.yml
            - name: storage-logs
              mountPath: /app/logs
          imagePullPolicy: Always
      restartPolicy: Always

通过上面的yaml配置内容可以发现，containers节点下引入了2个镜像，这样做的目的就是为了将应用产生的日志文件挂载同一个pod中，filebeat再监听到文件中的内容将日志内容发送至logstash。

  1. pod中挂载一个空的卷名称：storage-logs。

  2. 应用app中将/app/logs文件夹下的文件挂载到卷名称storage-logs下。

  3. filebeat容器中再使用到卷storage-logs，挂载到目录/app/logs下。

  4. filebeat将监听到自身容器中这个目录下文件内容。

filebeat配置文件内容如下，使用k8s的ConfigMap配置引入

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
kind: ConfigMap
apiVersion: v1
metadata:
  name: filebeat-app-configmap
  namespace: default
data:
  filebeat-config: |
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
      - /app/logs/app.log
      fields:
        app_id: app
      multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}'
      multiline.negate: false
      multiline.match: after  
      
    output.logstash:
      hosts: ["logstashapp:5044"]
      enabled: true

multiline：以时间格式为一行解析日志内容。

output.logstash：将日志信息发送至logstashapp:5044这个地址中。

由于filebeat的高版本不再支持多个输出，因此官方推荐使用logstash输出到多个地方，例如输出到文件、ES、或者其他数据库中。

logstash的yaml配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
kind: Deployment
apiVersion: apps/v1
metadata:
  name: logstashapp
  namespace: default
  labels:
    k8s-app: logstashapp
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: logstashapp
  template:
    metadata:
      name: logstashapp
      creationTimestamp: null
      labels:
        k8s-app: logstashapp
    spec:
      volumes:
        - name: logstash
          configMap:
            name: logstash
            items:
              - key: logstash-config
                path: logstash.conf
              - key: logstash-yml-config
                path: logstash.yml
            defaultMode: 420
        - name: logstash-kafka
          configMap:
            name: logstash-kafka
            items:
              - key: kafka_client_jaas.conf
                path: kafka_client_jaas.conf
            defaultMode: 420
      containers:
        - name: logstashapp
          image: 'logstash:6.8.8'
          command:
            - logstash
            - '-f'
            - /usr/share/logstash/config/logstash.conf
          ports:
            - name: logport
              containerPort: 5044
              protocol: TCP
          env:
            - name: ES_JAVA_OPTS
              value: '-Xms512m -Xmx512m'
            - name: TZ
              value: Asia/Shanghai
          resources:
            limits:
              cpu: '4'
              memory: 2Gi
            requests:
              cpu: '4'
              memory: 2Gi
          volumeMounts:
            - name: logstash
              mountPath: /usr/share/logstash/config/logstash.conf
              subPath: logstash.conf
            - name: logstash
              mountPath: /usr/share/logstash/config/logstash.yml
              subPath: logstash.yml
            - name: logstash-kafka
              mountPath: /usr/share/logstash/config/kafka_client_jaas.conf
              subPath: kafka_client_jaas.conf
          imagePullPolicy: IfNotPresent
      restartPolicy: Always

这个版本的logstash默认支持kafka的输出以及kafka的鉴权。这里设置的replicas实例为1，CPU为4，如果日志采集压力过大，可能收集日志的延迟并且CPU满载，根据实际情况增加实例节点，并且增加CPU的个数。

logstash配置文件使用ConfigMap挂载

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
kind: ConfigMap
apiVersion: v1
metadata:
  name: logstash
  namespace: default
data:
  logstash-config: |
    input {
      beats {
         port => 5044
         client_inactivity_timeout => 36000
      }
    }

    filter {
        ruby {
                # 新增索引日期,解决地区时差问题
                code => "event.set('index_date', event.get('@timestamp').time.localtime + 8*60*60)"
            }
         mutate {
                 convert => ["index_date", "string"]
                 gsub => ["index_date", "T([\S\s]*?)Z", ""]
                 gsub => ["index_date", "-", "."]
                 gsub => ["message", "(?m)(.{100000}).*", "\1"]
            }
    }

    output {
               elasticsearch {
                  hosts => ["http://xx.xx.xx.xx:34095"]
                  index => "app-log-%{index_date}"
                  user => "elastic"
                  password => "xxxxx"
               }
               file {
                  path => "/tmp/app-%{index_date}.log"
                  codec => line {
                    format => "%{message}"
                  }
                }
                kafka {
                  bootstrap_servers => "xx.xx.xx.xx:32049"
                  security_protocol => "SASL_PLAINTEXT"
                  sasl_mechanism => "SCRAM-SHA-256"
                  jaas_path => "/usr/share/logstash/config/kafka_client_jaas.conf"
                  topic_id => ["app-log"]
                  codec => json
                }
              
    }
  logstash-yml-config: |
    http.host: "0.0.0.0"
    http.port: 9600
    xpack.monitoring.elasticsearch.hosts: http://xx.xx.xx.xx:34095
    xpack.monitoring.elasticsearch.username: "elastic"
    xpack.monitoring.elasticsearch.password: "xx"

kafka鉴权配置文件，使用ConfigMap配置

1
2
3
4
5
6
7
8
9
10
11
12
13
kind: ConfigMap
apiVersion: v1
metadata:
  name: logstash-kafka
  namespace: default
data:
  kafka_client_jaas.conf: |
    KafkaClient {
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="xxxx"
      password="xxxx";
    };

根据实际情况修改鉴权的方式。

上一页
2023-09-03 20:07:08
下一页