使用filebeat收集iis,apache日志，通过grafana显示

2019-04-18

> 本文介绍如何使用 filebeat 收集 iis, apache 的日志送往 elk, 然后通过 Grafana 监控访问状况

## 安装 filebeat

Linux可以参考：

    https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

win可以在下面地址下载压缩包：

    https://www.elastic.co/downloads/beats/filebeat

## 配置 filebeat module

通过package安装的filebeat，档案位置在：

    /etc/filebeat

启用 modules

    filebeat modules enable apache
    filebeat modules enable iis

完成之后，可以在 modules.d 目录下找到 module 的配置文件：iis.yml, apache.yml，可以配置log路径

示例：

```yaml
# Module: apache
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-module-apache.html

- module: apache
  # Access logs
  access:
    enabled: true
    var.paths: ["/var/log/apache2/access.log*","/etc/filebeat/apachelog/access.log*"]
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    # var.paths: ["/proj/mtk40329/filebeat/apachelog/*.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
```



## 配置 filebeat

filebeat的配置文件为 filebeat.yml

需要配置的参数主要为：
* reload.enabled: true，可以重新读取配置信息
* output.elasticsearch.hosts，elk地址
* setup.ilm，配置index名称及rollover后缀

```yaml
#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xxx:9200"]

#============================== Setup ILM =====================================

# Configure Index Lifecycle Management Index Lifecycle Management creates a
# write alias and adds additional settings to the template.
# The elasticsearch.output.index setting will be replaced with the write alias
# if ILM is enabled.

# Enabled ILM support. Valid values are true, false, and auto. The beat will
# detect availabilty of Index Lifecycle Management in Elasticsearch and enable
# or disable ILM support.
setup.ilm.enabled: auto

# Configure the ILM write alias name.
setup.ilm.rollover_alias: "access.xxxweb"

# Configure rollover index pattern.
setup.ilm.pattern: "001"

```

对于支持ILM(index lifecycle management)的elk，想要更改filebeat创建的index名称，配置output.elasticsearch.index是不起作用的，需要修改ILM的配置。

## 启动 filebeat

可以通过 service filebeat start，或者 filebeat run 来启动。

## 配置 Grafana

配置好 Grafana 的 elk search source 源之后，就可以通过 dashboard 查看 filebeat 送往 elk 的日志了。

### apache

获取 apache 的 pv:

    _index:access.xxxweb* AND url.original:*.html AND NOT http.response.status_code:401
    
获取 apache 的 uv:

    _index:access.xxxweb* AND url.original:*.html AND NOT http.response.status_code:401

metric修改为 source.ip 的unique count

获取热门page：

    _index:access.xxxweb* AND url.original:*.html AND NOT http.response.status_code:401

groupby: Terms, field: url.original, top: 10

### iis

获取 iis 的 pv:

    _index:access.xxxweb* AND url.path:*.html AND NOT http.response.status_code:401
    
获取 iis 的 uv:

    _index:access.xxxweb* AND url.path:*.html AND NOT http.response.status_code:401

metric修改为 user.name 的unique count

获取热门page：

    _index:access.xxxweb* AND url.path:*.html AND NOT http.response.status_code:401

groupby: Terms, field: url.path, top: 10

## 参考

* [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)
* [ILM](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html)
* [elasticsearch-output](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html)

本文为博主原创，如需转载，请注明出处：http://www.supperxin.com

返回博客列表