实践logstash

Posted on | In | Visitors

Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.

While Logstash originally drove innovation in log collection, its capabilities extend well beyond that use case. Any type of event can be enriched and transformed with a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater volume and variety of data.

Logstash Reference

样例

从标准输入采集数据，经过grok过滤处理，最后解码输出到控制台。适合用于测试验证。

logstash.conf：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
input {
  stdin {
    type => stdin
  }
}

filter {
  grok {
    match => { "message" => "%{NUMBER:duration} %{IP:client}" }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

运行：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ bin/logstash -f config/logstash-sample.conf 
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /home/angi/software/logstash/logs which is now configured via log4j2.properties
[2017-11-07T14:10:11,636][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-11-07T14:10:11,665][INFO ][logstash.pipeline        ] Pipeline main started
The stdin plugin is now waiting for input:
[2017-11-07T14:10:11,711][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
# 下面是输入
23 192.168.2.2
# 下面是输出
{
      "duration" => "23",
    "@timestamp" => 2017-11-07T06:10:36.318Z,
      "@version" => "1",
          "host" => "angi-deepin",
        "client" => "192.168.2.2",
       "message" => "23 192.168.2.2",
          "type" => "stdin"
}

input

stdin

1
2
3
4
5
input {
  stdin {
    type => stdin
  }
}

beat

1
2
3
4
5
input {
  beats {
    port => 5044
  }
}

filter

Filter plugins

grok

1
2
3
4
5
filter {
  grok {
    match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}]\s+\[%{GREEDYDATA:thread}]\s+\[%{LOGLEVEL:loglevel}]\s+\[%{GREEDYDATA:traceId}]\s+\[%{USERNAME:logger}]\s+-\s+%{GREEDYDATA:msg}" }
  }
}

样例：从filebeat来的数据经过logstash的grok处理后的结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
{
          "timestamp" : "2021-09-03 12:18:51.723",
          "msg" : """通知转发: {"actionType":2,"code":200,"deviceId":"868089050237340","gridNo":"02&01","reqNo":"200000000561320210519"}""",
          "@version" : "1",
          "log" : {
            "file" : {
              "path" : "/app/third-party-service/log/zhkj-webapi.log"
            },
            "offset" : 351812
          },
          "message" : """[2021-09-03 12:18:51.723] [http-nio-8080-exec-1] [INFO] [] [c.z.k.a.m.i.service.CabinetAsynNotify] - 通知转发: {"actionType":2,"code":200,"deviceId":"868089050237340","gridNo":"02&01","reqNo":"200000000561320210519"}""",
          "ecs" : {
            "version" : "1.5.0"
          },
          "host" : {
            "containerized" : false,
            "os" : {
              "platform" : "centos",
              "version" : "7 (Core)",
              "kernel" : "3.10.0-957.el7.x86_64",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "family" : "redhat"
            },
            "architecture" : "x86_64",
            "mac" : [
              "52:54:00:34:d7:d4"
            ],
            "name" : "dev-1-202",
            "ip" : [
              "192.168.1.202",
              "fe80::39ca:ea1d:2de2:c04",
              "fe80::c31:cdf:e1b9:c9fe",
              "fe80::99df:f4da:4cfa:c141"
            ],
            "hostname" : "dev-1-202",
            "id" : "4019616339ee41f2a235a10f31397ac7"
          },
          "@timestamp" : "2021-09-03T04:18:52.572Z",
          "loglevel" : "INFO",
          "logger" : "c.z.k.a.m.i.service.CabinetAsynNotify",
          "thread" : "http-nio-8080-exec-1",
          "agent" : {
            "type" : "filebeat",
            "version" : "7.7.1",
            "ephemeral_id" : "b1d610c2-188f-4515-b03e-e836bc78b157",
            "hostname" : "dev-1-202",
            "id" : "4c3f3598-cb25-4098-979d-b1dbfeb54e54"
          },
          "input" : {
            "type" : "log"
          },
          "tags" : [
            "beats_input_codec_plain_applied"
          ]
        },
        "sort" : [
          1630642732572
        ]
      }

output

stdout

1
2
3
4
5
output {
  stdout {
    codec => rubydebug
  }
}

elasticsearch

1
2
3
4
5
6
7
8
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}