# thanos-cluster

**Repository Path**: mario0718/thanos-cluster

## Basic Information

- **Project Name**: thanos-cluster
- **Description**: kubernetes基础环境下跨区域分布式thanos监控集群，统一检索promQL，独立规则和告警触发
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-05-12
- **Last Updated**: 2023-05-25

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

## 一、Thanos 组件介绍：
### 方案选型
> Sidecar or Receiver
目前官方的架构图用的 Sidecar 方案，Receiver 是一个暂时还没有完全发布的组件。通常来说，Sidecar 方案相对成熟一些，最新的数据存储和计算（比如聚合函数）比较 “分布式”，更加高效也更容易扩展。

![20230512_092601_87](image/20230512_092601_87.png)

Receiver 方案是让 Prometheus 通过 remote wirte API 将数据 push 到 Receiver 集中存储(同样会清理过期数据):

![20230512_092742_27](image/20230512_092742_27.png)

那么该选哪种方案呢？我的建议是：
- 1. 如果你的 Query 跟 Sidecar 离的比较远，比如 Sidecar 分布在多个数据中心，Query 向所有 Sidecar 查数据，速度会很慢，这种情
况可以考虑用 Receiver，将数据集中吐到 Receiver，然后 Receiver 与 Query 部署在一起，Query 直接向 Receiver 查最新数据，提升查询性能。
- 2. 如果你的使用场景只允许 Prometheus 将数据 push 到远程，可以考虑使用 Receiver。比如 IoT 设备没有持久化存储，只能将数据push 到远程。
- 3. 此外的场景应该都尽量使用 Sidecar 方案。

> 评估是否需要 Ruler
- Ruler 是一个可选组件，原则上推荐尽量使用 Prometheus 自带的 rule 功能（生成新指标+告警），这个功能需要一些 Prometheus 最新数据，直接使用 Prometheus 本机 rule 功能和数据，性能开销相比 Thanos Ruler 这种分布式方案小得多，并且几乎不会出错，Thanos Ruler 由于是分布式，所以更容易出错一些。
- 如果某些有关联的数据分散在多个不同 Prometheus 上，比如对某个大规模服务采集做了分片，每个 Prometheus 仅采集一部分 Endpoint 的数据，对于 record 类型的 rule（生成的新指标），还是可以使用 Prometheus 自带的 rule 功能，在查询时再聚合一下就可以（如果可以接受的话）；对于 alert 类型的 rule，就需要用 Thanos Ruler 来做了，因为有关联的数据分散在多个 Prometheus 上，用单机数据去做 alert 计算是不准确的，就可能会造成误告警或不告警。
> 评估是否需要 Store Gateway 与 Compact
- Store 也是一个可选组件，也是 Thanos 的一大亮点的关键：数据长期保存。
评估是否需要 Store 组件实际就是评估一下自己是否有数据长期存储的需求，比如查看一两个月前的监控数据。如果有，那么 Thanos 可以将数据上传到对象存储保存。Thanos 支持以下对象存储:
  - Google Cloud Storage
  - AWS/S3
  - Azure Storage Account
  - OpenStack Swift
  - Tencent COS
  - AliYun OSS

- 通常使用了对象存储来长期保存数据不止要安装 Store Gateway，还需要安装Compact 来对对象存储里的数据进行压缩与降采样，这样可以提升查询大时间范围监控数据的性能。注意：Compact 并不会减少对象存储的使用空间，而是会增加，增加更长采样间隔的监控数据，这样当查询大时间范围的数据时，就自动拉取更长时间间隔采样的数据以减少查询数据的总量，从而加快查询速度（大时间范围的数据不需要那么精细），当放大查看时（选择其中一小段时间），又自动选择拉取更短采样间隔的数据，从而也能显示出小时间范围的监控细节。

### Thanos Sidecar
这里以 Thanos 最新版本为例，选择 Sidecar 方案，介绍各个组件的 Kubernetes yaml 定义方式并解释一些重要细节。Thanos通过Sidecar进程与已有的Promerheus集成，Sidecar与Prometheus运行在同一个主机或Pod中。Sidecar的作用是将Prometheus的本地数据备份到对象存储中，并通过gRPC API为其他Thanos组件提供指标查询能力。集群模式使用 Prometheus 自带的 rule 功能(生成新指标+告警)，这个功能需要 Prometheus 最新数据，直接
使用 Prometheus 本机 rule 功能和数据，性能开销相比 Thanos Ruler 这种分布式方案小得多。

![20230511_142433_64](image/20230511_142433_64.png)

### 组件功能描述:

- 查询网关(Querier/Query)：实现了Prometheus API，与汇集底层组件(如边车组件Sidecar，或是存储网关Store Gateway)的数据。使用Thanos的全局查询层来一次性查询多个Prometheus的数据。Query组件是无状态的，可以任意**拓展副本数**。Query在连接到Sidecar后，会自动检测某个PromQL查询需要使用哪些Prometheus。

      Thanos Querier还实现了Prometheus的官方HTTP API (http://xxxx:9090)，因此可以直接作为Gafana的数据源使用。

- 存储网关(Store Gateway)：因为Sidecar会将数据备份到对象存储中，因此我们将Prometheus的本地数据保留时长设置得短一些以减少磁盘空间占用。所以我们还需要一个组件来查询对象存储minio中的历史数据，这个组件就是Store Gateway，同Sidecar一样，它也提供了StoreAPI接口，并且需要被Query组件调用。

- 规则组件(Ruler/Rule)：针对数据进行评估和报警，使用集群Prometheus自带的 rule 功能(生成新指标+告警)，并统一使用altermanager触发告警通知。

- 对象存储组件(开源minio、阿里oss、腾讯cos):存储prometheus以tsdb存储形式的数据文件以及告警规则。

- 压缩器(Compactor)：将对象存储中的数据进行压缩和下采样以及设置数据保存时间，并不是将数据进行压缩从而降低磁盘使用空间，反而会使用更多的空间，该组件的主要用途是用于提高跨长时间度的查询速度；它也不会直接删除云存储中的数据，只是会将过期的数据进行标记，并存储在 deletion-mark.json 文件中。

大致的架构图大概如下:

![20230512_101257_20](image/20230512_101257_20.png)

## 二、安装组件
### 场景:
- 在两个不同区域的kubernetes集群下，各自创建monitor命名空间;详细见如下列表:

| 区域 | 组件 |
| ---- | ---- |
|集群A |  (alertmanager、grafana) Prometheus-operator、Thanos-Query、Store-Gateway、minio、Thanos-compactor|
|集群B |  Prometheus-operator、Thanos-Query、Store-Gateway|
| ... |  Prometheus-operator、Thanos-Query、Store-Gateway|

    说明:如果条件允许的话，AlertManager、Grafana、Thanos-compactor、Minio等组件可以安装在一个专门的监控集群中，这里因为资源有限，就将这些资源安装在了A集群中；其他集群仅安装Prometheus-operator、Thanos-Query、Store-Gateway；ruler+altermanager分别使用集群内prometheus自带的规则和告警功能。


### 2.1、集群A部署

#### 2.1.1 对象存储
目前 thanos 支持大部分云厂商的对象存储服务，具体使用请参考 thanos 对象存储 ，这里使用 minio 代替 S3 对象存储

- 前提安装有glusterfs分布式存储或者云储存，这里假设以安装有3台glusterfs分布式存储。

- 安装GlusterFS subdir external provisioner动态存储挂载组件

  > 这里minio使用yaml清单安装
       >> minio-sts.yaml   //endpoints绑定glusterfs服务端地址
       >> ingress-minio.yaml //制定minio对外暴露域名

     >> kubectl执行安装后，登录 minio 创建一个名为thanos的 bucket桶

- 通过ingress-nginx四层代理方式暴露minio端口，编辑修改Ingress-nginx配置
```
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  "9000": monitoring/thanos-minio:9000
```

- 在每个集群中都创建一个存储 secret ，这里集群A创建文件
thanos-store-secret.yaml
```
   kubectl apply -f thanos-store-secret.yaml
```
  文件内容:
```
---
apiVersion: v1
kind: Secret
metadata:
  name: thanos-store
  namespace: monitoring
type: Opaque
stringData:
  thanos-store-minio.yaml: |-
    type: s3
    config:
      bucket: thanos
      endpoint: minio:9000
      insecure: true
      signature_version2: false
      access_key: YOURACCESSKEY  //修改为实际的access_key
      secret_key: YOURSECRETKEY  //修改为实际的secret_key
```
       (1)比如使用腾讯云 COS 来存储:
```
    apiVersion: v1
    kind: Secret
    metadata:
      name: thanos-store
      namespace: monitoring
    type: Opaque
    stringData:
      thanos-store-minio.yaml: |
        type: COS
        config:
          bucket: "thanos"
          region: "ap-singapore"
          app_id: "12*******5"
          secret_key: "tsY***************************Edm"
          secret_id: "AKI******************************gEY"
```
       (2)或者使用阿里云 OSS 存储:
```
    apiVersion: v1
    kind: Secret
    metadata:
      name: thanos-store
      namespace: monitoring
    type: Opaque
    stringData:
      thanos-store-minio.yaml: |
        type: ALIYUNOSS
        config:
          endpoint: "oss-cn-hangzhou-internal.aliyuncs.com"
          bucket: "thanos"
          access_key_id: "LTA******************KBu"
          access_key_secret: "oki************************2HQ"
```
#### 2.1.2、Prometheus-operator

按照prometheus-operator使用kube-prometheus进行安装完成后，需要修改的文件 prometheus-prometheus.yaml

> (1)添加编辑如下内容
```
  externalLabels:
    cluster: cluster-a
    # 查询的数据中，会包含cluster标签，用于区分集群数据
  thanos:
    image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
      requests:
        cpu: 100m
        memory: 500Mi
    version: v0.28.1
    objectStorageConfig:
      key: thanos-store-minio.yaml //引用minio的访问配置
      name: thanos-store
```
> (2)注释掉如下内容
```
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
```
> (3)执行安装
```
kubectl delete -f prometheus-prometheus.yaml
kubectl apply -f prometheus-prometheus.yaml
```
> 说明:
>> 1.因为不需要Prometheus和AlertManager直接通信，后面改用Ruler组件;

>> 2.修改执行以上文件资源即可，Thanos的Sidecar组件就会自动部署上;

>> 3.配置AlertManager邮件告警或集成企业微信告警，可参考之前的文章：prometheus-operator 添加自定义告警。

#### 2.1.3、Store Gateway
- 主要负责查询对象存储minio中的历史数据。编辑文件 thanos-store-gateway-statefulSet.yaml,创建store-gateway组件。

> thanos-store-gateway-statefulSet.yaml
```
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: object-store-gateway
    app.kubernetes.io/instance: thanos-store
    app.kubernetes.io/name: thanos-store
    app.kubernetes.io/version: v0.28.1
  name: thanos-store
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/component: object-store-gateway
      app.kubernetes.io/instance: thanos-store
      app.kubernetes.io/name: thanos-store
  serviceName: thanos-store
  template:
    metadata:
      labels:
        app.kubernetes.io/component: object-store-gateway
        app.kubernetes.io/instance: thanos-store
        app.kubernetes.io/name: thanos-store
        app.kubernetes.io/version: v0.28.1
    spec:
      containers:
      - args:
        - store
        - --log.level=info
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901 //供thanos-query调用
        - --http-address=0.0.0.0:10902
        - --objstore.config-file=$(OBJSTORE_CONFIG)
        - "--index-cache-size=500MB"
        - "--chunk-pool-size=500MB"
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos-store-minio.yaml //引用minio的访问配置
              name: thanos-store
        image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
        livenessProbe:
          failureThreshold: 8
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
        name: thanos-store-gateway
        resources:
          limits:
            cpu: 500m
            memory: 2000Mi
          requests:
            cpu: 100m
            memory: 256Mi
        ports:
        - containerPort: 10901
          name: grpc
        - containerPort: 10902
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          periodSeconds: 5
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/thanos/store
          name: data
          readOnly: false
      terminationGracePeriodSeconds: 120
      volumes: []
  volumeClaimTemplates:
  - metadata:
      labels:
        app.kubernetes.io/component: object-store-gateway
        app.kubernetes.io/instance: thanos-store
        app.kubernetes.io/name: thanos-store
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: "glusterfs-client"
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
```
- 创建store-gateway服务，用于为thanos-query组件提供查询接口
> thanos-store-gateway-service.yaml
```
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: object-store-gateway
    app.kubernetes.io/instance: thanos-store
    app.kubernetes.io/name: thanos-store
    app.kubernetes.io/version: v0.28.1
  name: thanos-store-gateway
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: grpc
    port: 10901
    targetPort: 10901
  - name: http
    port: 10902
    targetPort: 10902
  selector:
    app.kubernetes.io/component: object-store-gateway
    app.kubernetes.io/instance: thanos-store
    app.kubernetes.io/name: thanos-store
```
- 编辑文件 thanos-sidecar-service.yaml，将Prometheus数据通过sidecar组件暴露给thanos-query组件查询
```
apiVersion: v1
kind: Service
metadata:
  name: thanos-sidecar
  namespace: monitoring
  labels:
    app: thanos-sidecar
spec:
  clusterIP: None
  selector:
    prometheus: k8s
  ports:
  - name: grpc
    port: 10901
    targetPort: grpc
```
- 如下资源执行安装:
```
  kubectl apply -f thanos-store-gateway-statefulSet.yaml
  kubectl apply -f thanos-store-gateway-service.yaml
  kubectl apply -f thanos-sidecar-service.yaml
```
#### 2.1.4、Thanos Query
- 收集除本集群以外(集群B)开放的thanos-sidecar的storeAPI(grpc地址和grpc端口)
```
thanos-sd-storeapi-configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
  name: storeapi-target-config
  namespace: monitoring
data:
  thanos-sd-file.yaml: |-
    - targets:
      - x.x.x.x:10901
```
- 暴露集群B中sidecar grpc接口供集群A的thanos-querty提供访问，编辑Ingress-nginx配置
```
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  "10901": monitoring/thanos-sidecar:10901
```

- 编辑文件 thanos-query.yaml，创建双副本的Query组件
```
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
    app.kubernetes.io/version: v0.28.1
  name: thanos-query
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/component: query-layer
      app.kubernetes.io/instance: thanos-query
      app.kubernetes.io/name: thanos-query
  template:
    metadata:
      labels:
        app.kubernetes.io/component: query-layer
        app.kubernetes.io/instance: thanos-query
        app.kubernetes.io/name: thanos-query
        app.kubernetes.io/version: v0.28.1
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app.kubernetes.io/name
                      operator: In
                      values:
                        - thanos-query
                topologyKey: kubernetes.io/hostname
      tolerations:
      - key: "node.kubernetes.io/unreachable"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 2
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 2
      containers:
      - args:
        - query
        - --log.level=info
        - --query.partial-response
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --query.auto-downsampling
        - --query.replica-label=prometheus_replica
        - --store=dnssrv+_grpc._tcp.thanos-sidecar.monitoring.svc.cluster.local # thanos-query需要调grpc端口做监控数据查询,这里是集群A的thanos-sidecar暴露的服务
        - --store=dnssrv+_grpc._tcp.thanos-store-gateway.monitoring.svc.cluster.local # thanos-query需要调grpc端口做监控数据查询,这里是集群A的store-gateway暴露的服务
        - --store.sd-files=/etc/thanos/sd/thanos-sd-file.yaml  # thanos-query需要调grpc端口做监控数据查询,部署在集群外使用配置文件来做服务发现
        - --store.sd-interval=1m
        image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 9090
            scheme: HTTP
          periodSeconds: 30
        name: thanos-query
        volumeMounts:
        - name: thanos-sd-config
          mountPath: /etc/thanos/sd
        resources:
          requests:
            memory: "2Gi"
            cpu: "1"
          limits:
            memory: "2Gi"
            cpu: "1"
        env:
        - name: TZ
          value: Asia/Shanghai
        ports:
        - containerPort: 10901
          name: grpc
        - containerPort: 9090
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 9090
            scheme: HTTP
          periodSeconds: 5
        terminationMessagePolicy: FallbackToLogsOnError
      terminationGracePeriodSeconds: 120
      volumes:
      - name: thanos-sd-config
        configMap:
          name: storeapi-target-config
---
apiVersion: v1
kind: Service
metadata:
  name: thanos-query
  namespace: monitoring
  labels:
    app: thanos-query
spec:
  selector:
    app.kubernetes.io/instance: thanos-query
  ports:
  - name: http
    port: 9090
    targetPort: http
```
-  如下资源执行安装:
```
  kubectl apply -f thanos-query.yaml
  kubectl apply -f thanos-sd-storeapi-configmap.yaml
```
#### 2.1.5、thanos-alertmanager
- 编辑文件 thanos-alertmanagerConfig.yaml,创建alertmanager连接信息.
```
apiVersion: v1
kind: Secret
metadata:
  name: thanos-alert-config
  namespace: monitoring
type: Opaque
stringData:
  thanos-alert-config.yaml: |-
    alertmanagers:
    - static_configs:
      - alertmanager-main:9093
      scheme: http
      path_prefix: "/"
      timeout: 10s
      api_version: v1
```
-  如下资源执行安装:
```
  kubectl apply -f thanos-alertmanagerConfig.yaml
```
#### 2.1.6、thanos-ruler
- 编辑文件 thanos-ruler.yaml 创建ruler组件，使用prometheus-operator本地集群告警规则。
```
apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: thanos-ruler
  namespace: monitoring
  labels:
    app: thanos-ruler
spec:
  image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
  alertmanagersConfig:
    key: thanos-alert-config.yaml
    name: thanos-alert-config
  replicas: 1
  resources:
    requests:
      memory: 200Mi
    limits:
      memory: 1000Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules  #prometheus-operator本地告警规则
  queryEndpoints:
  - dnssrv+_http._tcp.thanos-query.monitoring.svc.cluster.local
  objectStorageConfig:
    key: thanos-store-minio.yaml
    name: thanos-store
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: "glusterfs-client"
        resources:
          requests:
            storage: 2Gi
```
-  如下资源执行安装:
```
  kubectl apply -f thanos-ruler.yaml
```
#### 2.1.7、thanos-compactor
- Compact 只能部署单个副本，因为如果多个副本都去对对象存储的数据做压缩和降采样的话，会造成冲突。
- thanos-compactor.yaml
```
apiVersion: v1
kind: Service
metadata:
  name: thanos-compact
  namespace: monitoring
  labels:
    app.kubernetes.io/name: thanos-compact
spec:
  ports:
  - name: http
    port: 10902
    targetPort: http
  selector:
    app.kubernetes.io/name: thanos-compact
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-compactor
  namespace: monitoring
  labels:
    app: thanos-compactor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-compactor
  serviceName: thanos-compactor
  template:
    metadata:
      labels:
        app: thanos-compactor
    spec:
      containers:
      - name: thanos
        image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
        args:
        - "compact"
        - "--log.level=debug"
        - "--data-dir=/var/thanos/store"
        - "--objstore.config-file=$(OBJSTORE_CONFIG)"
        - "--retention.resolution-raw=90d"  #指定原始数据存放时长
        - "--retention.resolution-5m=180d"  #指定降采样到数据点 5 分钟间隔的数据存放时长
        - "--retention.resolution-1h=360d"   #指定降采样到数据点 1 小时间隔的数据存放时长，它们的数据精细程度递减，占用的存储空间也是递减，通常建议它们的存放时间递增配置（一般只有比较新的数据才会放大看，久远的数据通常只会使用大时间范围查询来看个大致，所以建议将精细程度低的数据存放更长时间）。
        - "--wait" #让 Compact 一直运行，轮询新数据来做压缩和降采样
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos-store-minio.yaml
              name: thanos-store
        ports:
        - name: http
          containerPort: 10902
        livenessProbe:
          httpGet:
            port: 10902
            path: /-/healthy
          initialDelaySeconds: 10
        readinessProbe:
          httpGet:
            port: 10902
            path: /-/ready
          initialDelaySeconds: 15
        volumeMounts:
        - mountPath: /var/thanos/store
          name: data
          readOnly: false
      terminationGracePeriodSeconds: 120
  volumeClaimTemplates:
  - metadata:
      labels:
        app.kubernetes.io/component: object-store-gateway
        app.kubernetes.io/instance: thanos-store
        app.kubernetes.io/name: thanos-store
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: "glusterfs-client"
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
```
-  如下资源执行安装:
```
  kubectl apply -f thanos-compactor.yaml
```
创建上面所有资源即可。

### 2.2 集群B部署
集群B通过加入到集群A来实现统一查询监控指标,再有其他集群想要接入的话，直接按照集群B的步骤操作即可。
#### 2.2.1、Prometheus-operator

按照集群A的步骤编辑: prometheus-prometheus.yaml

> (1)添加如下内容
```
    externalLabels:
      cluster: cluster-b
      # 查询的数据中，会包含cluster标签，用于区分集群数据,这里是集群b
    thanos:
      image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
      resources:
        limits:
          cpu: 500m
          memory: 500Mi
        requests:
          cpu: 100m
          memory: 500Mi
      version: v0.28.1
      objectStorageConfig:
        key: thanos-store-minio.yaml
        name: thanos-store
```
> (2)注释如下内容
```
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
```
按照之前的文章 prometheus-operator 安装 即可。

- 由于集群AB使用统一的对象存储minio，在每个集群中都需要创建secret来保存对象存储地址以及access-key和secret-key。
```
   kubectl apply -f thanos-store-secret.yaml
```
  文件内容:
```
---
apiVersion: v1
kind: Secret
metadata:
  name: thanos-store
  namespace: monitoring
type: Opaque
stringData:
  thanos-store-minio.yaml: |-
    type: s3
    config:
      bucket: thanos
      endpoint: x.x.x.x:9000  #连接集群A的minio对象存储,访问ingress开放的虚拟ip和四层代理端口
      insecure: true
      signature_version2: false
      access_key: YOURACCESSKEY  //修改为实际的access_key
      secret_key: YOURSECRETKEY  //修改为实际的secret_key
```
#### 2.2.2、Thanos Query
- 收集除本集群以外(集群A)开放的thanos-sidecar的storeAPI(grpc地址和grpc端口)
```
> thanos-sd-storeapi-configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
  name: storeapi-target-config
  namespace: monitoring
data:
  thanos-sd-file.yaml: |-
    - targets:
      - x.x.x.x:10901
```
- 按照集群A的步骤:集群B中prometheus需要通过sidecar才能暴露给A的thanos-query，不过使用Ingress-nginx四层代理方式集群A就能访问;编辑thanos-sidecar-service.yaml，将Prometheus数据通过sidecar组件暴露给集群A的query组件。
```
apiVersion: v1
kind: Service
metadata:
  name: thanos-sidecar
  namespace: monitoring
  labels:
    app: thanos-sidecar
spec:
  clusterIP: None
  selector:
    prometheus: k8s  #选择器指定prometheus的label名称
  ports:
  - name: grpc
    port: 10901
    targetPort: grpc
```
- 暴露集群B中sidecar grpc接口供集群A的thanos-querty提供访问，编辑Ingress-nginx配置
```
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  "10901": monitoring/thanos-sidecar:10901
```

- 将sidecar接口添加至thanos-query.yaml中;编辑文件 thanos-query.yaml，创建Query组件
```
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
    app.kubernetes.io/version: v0.28.1
  name: thanos-query
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/component: query-layer
      app.kubernetes.io/instance: thanos-query
      app.kubernetes.io/name: thanos-query
  template:
    metadata:
      labels:
        app.kubernetes.io/component: query-layer
        app.kubernetes.io/instance: thanos-query
        app.kubernetes.io/name: thanos-query
        app.kubernetes.io/version: v0.28.1
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app.kubernetes.io/name
                      operator: In
                      values:
                        - thanos-query
                topologyKey: kubernetes.io/hostname
      tolerations:
      - key: "node.kubernetes.io/unreachable"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 2
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 2
      containers:
      - args:
        - query
        - --log.level=info
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --query.partial-response
        - --query.replica-label=prometheus_replica
        - --query.auto-downsampling
        - --store=dnssrv+_grpc._tcp.thanos-sidecar.monitoring.svc.cluster.local   # thanos-query需要调grpc端口做监控数据查询,这里是集群B的thanos-sidecar暴露的服务
        - --store=dnssrv+_grpc._tcp.thanos-store-gateway.monitoring.svc.cluster.local   # thanos-query需要调grpc端口做监控数据查询,这里是集群B的store-gateway暴露的服务
        - --store.sd-files=/etc/thanos/sd/thanos-sd-file.yaml  # thanos-query需要调grpc端口做监控数据查询,部署在集群外使用配置文件来做服务发现
        - --store.sd-interval=1m
        image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 9090
            scheme: HTTP
          periodSeconds: 30
        name: thanos-query
        volumeMounts:
        - name: thanos-sd-config
          mountPath: /etc/thanos/sd
        resources:
          requests:
            memory: "2Gi"
            cpu: "1"
          limits:
            memory: "2Gi"
            cpu: "1"
        env:
        - name: TZ
          value: Asia/Shanghai
        ports:
        - containerPort: 10901
          name: grpc
        - containerPort: 9090
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 9090
            scheme: HTTP
          periodSeconds: 5
        terminationMessagePolicy: FallbackToLogsOnError
      terminationGracePeriodSeconds: 120
      volumes:
      - name: thanos-sd-config
        configMap:
          name: storeapi-target-config
---
apiVersion: v1
kind: Service
metadata:
  name: thanos-query
  namespace: monitoring
  labels:
    app: thanos-query
spec:
  selector:
    app.kubernetes.io/instance: thanos-query
  ports:
  - name: http
    port: 9090
    targetPort: http
```
#### 2.2.3、Store Gateway
- 主要负责查询对象存储minio中的历史数据。编辑文件 thanos-store-gateway-statefulSet.yaml,创建store-gateway组件。

> thanos-store-gateway-statefulSet.yaml
```
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: object-store-gateway
    app.kubernetes.io/instance: thanos-store
    app.kubernetes.io/name: thanos-store
    app.kubernetes.io/version: v0.28.1
  name: thanos-store
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/component: object-store-gateway
      app.kubernetes.io/instance: thanos-store
      app.kubernetes.io/name: thanos-store
  serviceName: thanos-store
  template:
    metadata:
      labels:
        app.kubernetes.io/component: object-store-gateway
        app.kubernetes.io/instance: thanos-store
        app.kubernetes.io/name: thanos-store
        app.kubernetes.io/version: v0.28.1
    spec:
      containers:
      - args:
        - store
        - --log.level=info
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901 //供thanos-query调用
        - --http-address=0.0.0.0:10902
        - --objstore.config-file=$(OBJSTORE_CONFIG)
        - "--index-cache-size=500MB"
        - "--chunk-pool-size=500MB"
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos-store-minio.yaml //引用minio的访问配置
              name: thanos-store
        image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
        livenessProbe:
          failureThreshold: 8
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
        name: thanos-store-gateway
        resources:
          limits:
            cpu: 500m
            memory: 2000Mi
          requests:
            cpu: 100m
            memory: 256Mi
        ports:
        - containerPort: 10901
          name: grpc
        - containerPort: 10902
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          periodSeconds: 5
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/thanos/store
          name: data
          readOnly: false
      terminationGracePeriodSeconds: 120
      volumes: []
  volumeClaimTemplates:
  - metadata:
      labels:
        app.kubernetes.io/component: object-store-gateway
        app.kubernetes.io/instance: thanos-store
        app.kubernetes.io/name: thanos-store
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: "glusterfs-client"
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
```
- 创建store-gateway服务，用于为thanos-query组件提供查询接口
> thanos-store-gateway-service.yaml
```
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: object-store-gateway
    app.kubernetes.io/instance: thanos-store
    app.kubernetes.io/name: thanos-store
    app.kubernetes.io/version: v0.28.1
  name: thanos-store-gateway
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: grpc
    port: 10901
    targetPort: 10901
  - name: http
    port: 10902
    targetPort: 10902
  selector:
    app.kubernetes.io/component: object-store-gateway
    app.kubernetes.io/instance: thanos-store
    app.kubernetes.io/name: thanos-store
```
- 编辑文件 thanos-sidecar-service.yaml，将Prometheus数据通过sidecar组件暴露给thanos-query组件查询
```
apiVersion: v1
kind: Service
metadata:
  name: thanos-sidecar
  namespace: monitoring
  labels:
    app: thanos-sidecar
spec:
  clusterIP: None
  selector:
    prometheus: k8s
  ports:
  - name: grpc
    port: 10901
    targetPort: grpc
```
- 如下资源执行安装:
```
  kubectl apply -f thanos-store-gateway-statefulSet.yaml
  kubectl apply -f thanos-store-gateway-service.yaml
  kubectl apply -f thanos-sidecar-service.yaml
```
#### 2.2.4、thanos-alertmanager
- 编辑文件 thanos-alertmanagerConfig.yaml,创建alertmanager连接信息.
```
apiVersion: v1
kind: Secret
metadata:
  name: thanos-alert-config
  namespace: monitoring
type: Opaque
stringData:
  thanos-alert-config.yaml: |-
    alertmanagers:
    - static_configs:
      - alertmanager-main:9093
      scheme: http
      path_prefix: "/"
      timeout: 10s
      api_version: v1
```
-  如下资源执行安装:
```
  kubectl apply -f thanos-alertmanagerConfig.yaml
```
#### 2.2.5、thanos-ruler
- 编辑文件 thanos-ruler.yaml 创建ruler组件，使用prometheus-operator本地集群告警规则。
```
apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: thanos-ruler
  namespace: monitoring
  labels:
    app: thanos-ruler
spec:
  image: registry.cn-shanghai.aliyuncs.com/pmg-register/thanos:v0.28.1
  alertmanagersConfig:
    key: thanos-alert-config.yaml
    name: thanos-alert-config
  replicas: 1
  resources:
    requests:
      memory: 200Mi
    limits:
      memory: 1000Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules  #prometheus-operator本地告警规则
  queryEndpoints:
  - dnssrv+_http._tcp.thanos-query.monitoring.svc.cluster.local
  objectStorageConfig:
    key: thanos-store-minio.yaml
    name: thanos-store
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: "glusterfs-client"
        resources:
          requests:
            storage: 2Gi
```
-  如下资源执行安装:
```
  kubectl apply -f thanos-ruler.yaml
```

至此基于thanos和Prometheus-operator的多集群监控就搭建完成了，再有其他集群想要接入的话，直接按照集群B的步骤操作即可。

查询监控数据时需要指定 Prometheus 数据源地址，由于我们使用了 Thanos 来做分布式，而 Thanos 关键查询入口就是 Query，所以我们需要将数据源地址指定为 Query 的地址，假如使用 Grafana 查询，进入 Configuration-Data Sources-Add data source，选择 Prometheus，指定本集群的Thanos-Query 的地址: http://thanos-query.monitoring.svc.cluster.local:9090。实现全局访问跨集群监控系统。

## 参考

- https://itnext.io/leveraging-consul-for-thanos-query-discovery-34212d496c88

- https://github.com/prometheus-operator/prometheus-operator/tree/master/example/thanos

- https://github.com/thanos-io/kube-thanos

- https://medium.com/@mail2ramunakerikanti/thanos-for-prometheus-f7f111e3cb75

- https://blog.csdn.net/sinat_35943558/article/details/116756443