告别笨重的 ELK，试试用 Loki 处理日志吧！

日志领域主流解决方案是 elk，但是这一套部署复杂、资源占用高，不适合个人开发者使用，如果你需要一个比命令行漂亮一些的日志查看工具，loki 就非常合适了。

promtail从 k8s/docker 中收集容器标准输出日志，发送到 loki 存储，grafana 在网页端对日志进行搜索和展示。非常轻量级，200MB 左右的内存就可以搞定。

安装

首先在一个安装了 docker 的服务器部署 loki、grafana 和收集本机 docker 容器日志的promtail：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
version: '3'
services:
  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    restart: unless-stopped
    networks:
      - npm
      - grafana
    # ports:
    #   - 3000:3000
    volumes:
      - ./volume/grafana/data/:/var/lib/grafana
  loki:
    image: grafana/loki
    container_name: loki
    restart: unless-stopped
    volumes:
      - ./volume/loki/config/:/etc/loki/
      - ./volume/loki/data/:/loki/
    # 修改loki默认配置文件路径
    command: -config.file=/etc/loki/loki.yml
    # ports:
    #   - 3100:3100
    networks:
      - npm
      - grafana
  promtail:
    image:  grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./volume/promtail/config/promtail.yml:/etc/promtail/docker-config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/docker-config.yaml
    networks:
      - grafana
      
networks:
  npm:
    external: true
  grafana:

这里出现了两个 docker 网络：npm 和 grafana，npm（nginx proxy manager） 是为了反向代理，grafana 是为了让这三个容器之间互通。

需要用到的 loki 配置文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://alertmanager:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
analytics:
 reporting_enabled: false

# 允许接收旧日志
limits_config:
  reject_old_samples: false

promtail 配置文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# https://grafana.com/docs/loki/latest/clients/promtail/configuration/
# https://docs.docker.com/engine/api/v1.41/#operation/ContainerList
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: flog_scrape
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'

这个 promtail 收集 docker 中所有容器的标准输出日志，并打上“container=容器名”的 lable 发送给 loki，以区分不同容器的日志。

安装好 docker 这一套以后，登录 grafana，添加 loki 数据源，在 explore 页面可以通过“{container=容器名}”的查询语句查看 docker 容器的日志。

k8s 的日志如何推送到 loki？

我在另外一台安装了 k8s 的服务器运行后端，我们还要把 k8s pod 的日志发送到 loki 里，便于统一查看。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
--- # Daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail-daemonset
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      serviceAccount: promtail-serviceaccount
      containers:
      - name: promtail-container
        image: grafana/promtail
        args:
        - -config.file=/etc/promtail/promtail.yaml
        env: 
        - name: 'HOSTNAME' # needed when using kubernetes_sd_configs
          valueFrom:
            fieldRef:
              fieldPath: 'spec.nodeName'
        volumeMounts:
        - name: logs
          mountPath: /var/log
        - name: promtail-config
          mountPath: /etc/promtail
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      volumes:
      - name: logs
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: promtail-config
        configMap:
          name: promtail-config
--- # configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: https://你的 loki 地址/loki/api/v1/push
      basic_auth:
        username: 你的 loki 用户名
        password: 你的 loki 密码

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          replacement: $1
          separator: /
          source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__

--- # Clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-clusterrole
rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - pods
    verbs:
    - get
    - watch
    - list

--- # ServiceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail-serviceaccount

--- # Rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-clusterrolebinding
subjects:
    - kind: ServiceAccount
      name: promtail-serviceaccount
      namespace: default
roleRef:
    kind: ClusterRole
    name: promtail-clusterrole
    apiGroup: rbac.authorization.k8s.io

这会在你k8s 的每一个节点上运行一个 promtail，收集pod日志发送到loki。

然后到 grafana 导入这个 dashboard，你就可以方便地查看 pod 日志了。

但是细心的同学应该能发现这个 crd 里有一些我没提到的参数（https 的 loki url，用户名，密码）。

这是因为我们不想让日志暴露在全世界面前，我们可以通过 npm 反向代理安全、便捷地实现这一点。

我的另一篇文章：轻松配置 https：Let‘s Encrypt 介绍及 Nginx Proxy Manager 实用操作教程 介绍了如何给 http 服务加上反向代理，并使用 basic auth 来保护接口。然后你就能得到你 loki 的 url、用户名和密码了。