强悍:【k8s 实战】Prometheus Operator 高级配置

　　强悍:【k8s 实战】Prometheus Operator 高级配置

　　以上我们学习了如何在Prometheus Operator下自定义一个监控选项，以及自定义告警规则的使用。那么我们还可以直接使用之前课程中的自动发现功能吗？如果我们的Kubernetes集群中有很多Services/Pod，是否需要一个一个创建对应的ServiceMonitor对象进行监控呢？这不会又麻烦了吧？

　　自动发现配置

　　为了解决上面的问题，Prometheus Operator为我们提供了额外的抓包配置来解决这个问题，我们可以添加额外的服务发现和自动监控的配置。和之前的自定义方式一样，我们希望在Prometheus Operator中自动发现和监控注解为prometheus.io/scrape=true的Service。我们之前定义的Prometheus的配置如下：

　　- job_name: 'kubernetes-service-endpoints'

kubernetes_sd_configs:

- role: endpoints

relabel_configs:

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]

action: keep

regex: true

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]

action: replace

target_label: __scheme__

regex: (https?)

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]

action: replace

target_label: __metrics_path__

regex: (.+)

- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]

action: replace

target_label: __address__

regex: ([^:]+)(?::\d+)?;(\d+)

replacement: $1:$2

- action: labelmap

regex: __meta_kubernetes_service_label_(.+)

- source_labels: [__meta_kubernetes_namespace]

action: replace

target_label: namespace

- source_labels: [__meta_kubernetes_service_name]

action: replace

target_label: service

- source_labels: [__meta_kubernetes_pod_name]

target_label: pod

action: replace

　　如果对上面的配置不是很熟悉，建议查看之前的Kubernetes公共资源对象监控介绍。要自动发现集群中的Service，我们需要在Service =true语句的注释区添加prometheus.io/scrape，将上述文件直接保存为prometheus-additional.yaml，然后创建对应的Secret对象通过这个文件：

　　$ kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

secret "additional-configs" created

　　注意，我们所有的操作都在 Prometheus Operator 源代码 contrib/kube-prometheus/manifests/ 目录下。

　　创建完成后，上述配置信息会被base64编码为key prometheus-additional.yaml对应的值：

　　$ kubectl get secret additional-configs -n monitoring -o yaml

apiVersion: v1

data:

prometheus-additional.yaml: LSBqb2JfbmFtZTogJ2t1YmVybmV0ZXMtc2VydmljZS1lbmRwb2ludHMnCiAga3ViZXJuZXRlc19zZF9jb25maWdzOgogIC0gcm9sZTogZW5kcG9pbnRzCiAgcmVsYWJlbF9jb25maWdzOgogIC0gc291cmNlX2xhYmVsczogW19fbWV0YV9rdWJlcm5ldGVzX3NlcnZpY2VfYW5ub3RhdGlvbl9wcm9tZXRoZXVzX2lvX3NjcmFwZV0KICAgIGFjdGlvbjoga2VlcAogICAgcmVnZXg6IHRydWUKICAtIHNvdXJjZV9sYWJlbHM6IFtfX21ldGFfa3ViZXJuZXRlc19zZXJ2aWNlX2Fubm90YXRpb25fcHJvbWV0aGV1c19pb19zY2hlbWVdCiAgICBhY3Rpb246IHJlcGxhY2UKICAgIHRhcmdldF9sYWJlbDogX19zY2hlbWVfXwogICAgcmVnZXg6IChodHRwcz8pCiAgLSBzb3VyY2VfbGFiZWxzOiBbX19tZXRhX2t1YmVybmV0ZXNfc2VydmljZV9hbm5vdGF0aW9uX3Byb21ldGhldXNfaW9fcGF0aF0KICAgIGFjdGlvbjogcmVwbGFjZQogICAgdGFyZ2V0X2xhYmVsOiBfX21ldHJpY3NfcGF0aF9fCiAgICByZWdleDogKC4rKQogIC0gc291cmNlX2xhYmVsczogW19fYWRkcmVzc19fLCBfX21ldGFfa3ViZXJuZXRlc19zZXJ2aWNlX2Fubm90YXRpb25fcHJvbWV0aGV1c19pb19wb3J0XQogICAgYWN0aW9uOiByZXBsYWNlCiAgICB0YXJnZXRfbGFiZWw6IF9fYWRkcmVzc19fCiAgICByZWdleDogKFteOl0rKSg/OjpcZCspPzsoXGQrKQogICAgcmVwbGFjZW1lbnQ6ICQxOiQyCiAgLSBhY3Rpb246IGxhYmVsbWFwCiAgICByZWdleDogX19tZXRhX2t1YmVybmV0ZXNfc2VydmljZV9sYWJlbF8oLispCiAgLSBzb3VyY2VfbGFiZWxzOiBbX19tZXRhX2t1YmVybmV0ZXNfbmFtZXNwYWNlXQogICAgYWN0aW9uOiByZXBsYWNlCiAgICB0YXJnZXRfbGFiZWw6IGt1YmVybmV0ZXNfbmFtZXNwYWNlCiAgLSBzb3VyY2VfbGFiZWxzOiBbX19tZXRhX2t1YmVybmV0ZXNfc2VydmljZV9uYW1lXQogICAgYWN0aW9uOiByZXBsYWNlCiAgICB0YXJnZXRfbGFiZWw6IGt1YmVybmV0ZXNfbmFtZQo=

kind: Secret

metadata:

creationTimestamp: 2018-12-20T14:50:35Z

name: additional-configs

namespace: monitoring

resourceVersion: "41814998"

selfLink: /api/v1/namespaces/monitoring/secrets/additional-configs

uid: 9bbe22c5-0466-11e9-a777-525400db4df7

type: Opaque

　　那么我们只需要在声明prometheus的资源对象文件中添加这个额外的配置即可：(prometheus-prometheus.yaml)

　　apiVersion: monitoring.coreos.com/v1

kind: Prometheus

metadata:

labels:

prometheus: k8s

name: k8s

namespace: monitoring

spec:

alerting:

<p>

alertmanagers:

- name: alertmanager-main

namespace: monitoring

port: web

baseImage: quay.io/prometheus/prometheus

nodeSelector:

beta.kubernetes.io/os: linux

replicas: 2

secrets:

- etcd-certs

resources:

requests:

memory: 400Mi

ruleSelector:

matchLabels:

prometheus: k8s

role: alert-rules

securityContext:

fsGroup: 2000

runAsNonRoot: true

runAsUser: 1000

additionalScrapeConfigs:

name: additional-configs

key: prometheus-additional.yaml

serviceAccountName: prometheus-k8s

serviceMonitorNamespaceSelector: {}

serviceMonitorSelector: {}

version: v2.5.0

</p>

　　添加完成后，直接更新prometheus的CRD资源对象：

　　$ kubectl apply -f prometheus-prometheus.yaml

prometheus.monitoring.coreos.com "k8s" configured

　　稍等片刻，可以到Prometheus的Dashboard查看配置是否生效：

　　在Prometheus Dashboard的配置页面下，我们可以看到已经有对应的配置信息，但是我们切换到targets页面，并没有找到对应的监控任务。查看Prometheus的Pod日志：

<p>$ kubectl logs -f prometheus-k8s-0 prometheus -n monitoring

level=error ts=2018-12-20T15:14:06.772903214Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:302: Failed to list *v1.Pod: pods is forbidden: User