fluentd收集k8s集群日志

2020-05-22

kubernetes

> 本文介绍如何使用fluentd在k8s集群做日志收集

## k8s日志收集方案

* Use a node-level logging agent that runs on every node.
* Include a dedicated sidecar container for logging in an application pod.
  * Streaming sidecar container
  * Sidecar container with a logging agent
* Push logs directly to a backend from within an application.


 | node-level logging agent                                                                                                                                       | Streaming sidecar container                                                                                                                                                  | Sidecar container with a logging agent                                                                                                                               |
 | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | ![node-agent](https://d33wubrfki0l68.cloudfront.net/2585cf9757d316b9030cf36d6a4e6b8ea7eedf5a/1509f/images/docs/user-guide/logging/logging-with-node-agent.png) | ![streaming-sidecar](https://d33wubrfki0l68.cloudfront.net/5bde4953b3b232c97a744496aa92e3bbfadda9ce/39767/images/docs/user-guide/logging/logging-with-streaming-sidecar.png) | ![sidecar-agent](https://d33wubrfki0l68.cloudfront.net/d55c404912a21223392e7d1a5a1741bda283f3df/c0397/images/docs/user-guide/logging/logging-with-sidecar-agent.png) |



|              | node-level logging agent | Streaming sidecar container | Sidecar container with a logging agent |
| ------------ | ------------------------ | --------------------------- | -------------------------------------- |
| kubectl logs | 支持                     | 支持                        | 不支持                                 |
| 灵活度       | 低                       | 中                          | 高                                     |
| 侵入性       | 低                       | 中                          | 高                                     |
| 资源消耗     | 低                       | 中                          | 高                                     |

## node-level logging agent: fluentd

用k8s的daemonset可以方便的在每个node上安装fluent agent，fluent提供了示例仓库：[fluentd-kubernetes-daemonset](https://github.com/fluent/fluentd-kubernetes-daemonset)

参考这一只来创建daemonset部署fluentd：https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch-rbac.yaml

```yaml
#@ load("@ytt:data", "data")

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-es
  namespace: logging
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-es
  namespace: logging
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-es
  apiGroup: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-es
  namespace: logging
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-es
  template:
    metadata:
      labels:
        k8s-app: fluentd-es
        kubernetes.io/cluster-service: "true"
      #! 此注释确保如果节点被驱逐，fluentd不会被驱逐，支持关键的基于 pod 注释的优先级方案。
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: fluentd-es
      containers:
      - name: fluentd-es
        image: #@ data.values.image
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor -q
        resources:
          limits:
            memory: 240Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config-volume
          mountPath: /fluentd/etc/
      nodeSelector:
        node-role.kubernetes.io/worker: "true"
      tolerations:
      - operator: Exists
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config-volume
        configMap:
          name: fluentd-config
```

配置configmap，作为配置文件挂载到fluentd：

```yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-config
  namespace: logging
data:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>
  kubernetes.conf: |-

  fluent.conf: |-
    <source>
      @id fluentd-containers.log
      @type tail                              # Fluentd 内置的输入方式，其原理是不停地从源文件中获取新的日志。
      path /var/log/containers/*.log          # 挂载的服务器Docker容器日志地址
      pos_file /var/log/es-containers.log.pos
      tag kubernetes.*                        # 设置日志标签
      read_from_head true
      <parse>                                 # 多行格式化成JSON
        @type multi_format                    # 使用 multi-format-parser 解析器插件
        <pattern>
          format json                         # JSON解析器
          time_key time                       # 指定事件时间的时间字段
          time_format %Y-%m-%dT%H:%M:%S.%NZ   # 时间格式
        </pattern>
      </parse>
    </source>

    # 添加 Kubernetes metadata 数据
    <filter kubernetes.**>
      @id filter_kubernetes_metadata
      @type kubernetes_metadata
    </filter>

    # 只保留具有logging=1标签的Pod日志
    <filter kubernetes.**>
      @id filter_log
      @type grep
      <regexp>
        key $.kubernetes.labels.logging
        pattern ^1$
      </regexp>
    </filter>

    # 删除一些多余的属性
    <filter kubernetes.**>
      @type record_transformer
      remove_keys $.docker.container_id,$.kubernetes.container_image_id,$.kubernetes.pod_id,$.kubernetes.namespace_id,$.kubernetes.master_url,$.kubernetes.labels.pod-template-hash
    </filter>

    <match kubernetes.**>
      @id elasticsearch
      @type elasticsearch
      @log_level debug
      include_tag_key true
      hosts [hosts]
      user [user]
      password [password]
      logstash_format true
      logstash_prefix [prefix]  # 设置 index 前缀为 k8s
      type_name doc
      request_timeout    30s
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        queue_limit_length 8
        overflow_action block
      </buffer>
    </match>
```

后续当configmap更新时，会自动更新挂载到容器中的文件，参考[mounted-configmaps-are-updated-automatically](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically)

创建自定义镜像，安装需要的插件：

```Dockerfile
FROM fluent/fluentd-kubernetes-daemonset:v1.10.4-debian-elasticsearch7-1.0

# 插件用于添加k8s的一些metadata信息到log中
RUN gem install fluent-plugin-kubernetes_metadata_filter
```

.dockerignore

```
Dockerfile
.dockerignore
README.md
*.yaml
*.sh
*.VERSION*
k8s/
```

### 部署脚本

创建部署脚本deploy.sh

```bash
#!/bin/bash
set -e

env="$1"
registryStage=""
registryProduction=""
versionFile=".VERSION"

if [ -z "$env" ]; then
    env="stage"
    registry=$registryStage
elif [ $env == 'production' ]; then
    registry=$registryProduction
fi

# generate version
bash version.sh $versionFile
version=`cat $versionFile`
img=$registry/fluentd:$version

echo "deploy: $env, version: $version, img: $img"

docker build . -t $img
docker push $img

echo "deploy configmap"
kubectl apply -f k8s/fluentd-configmap-$env.yaml

echo "deploy daemon set"
ytt -f k8s/fluentd-daemonset.yaml -f k8s/values.yaml --data-value image=$img | kubectl apply -f-
```

版本控管脚本version.sh

```bash
#!/bin/bash
set -e

versionFile="$1"
# if versionFile not given, set default file name
if [ -z "$versionFile" ]; then
    versionFile=".VERSION"
fi

# if file not exists, set default version
if [ ! -f "$versionFile" ]; then
    echo "0.0.0" > "$versionFile"
fi

echo "bump version"
# using treeder/bump to bump version
docker run --rm -v "$PWD":/app treeder/bump --filename "$versionFile"
```

## 参考

* [logging](https://kubernetes.io/docs/concepts/cluster-administration/logging)
* [configure-pod-configmap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/)
* [在 Kubernetes 上搭建 EFK 日志收集系统](https://www.qikqiak.com/post/install-efk-stack-on-k8s/)

本文为博主原创，如需转载，请注明出处：http://www.supperxin.com

返回博客列表