# monitor-agent

**Repository Path**: yi-junquan/monitor

## Basic Information

- **Project Name**: monitor-agent
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-10-31
- **Last Updated**: 2025-02-14

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Monitor agent

monitor-agent is a complementary  tool to node-exporter for monitoring more granular metrics

## Usage 

### Running monitor agent
```
./monitor-agent -h
monitor-agent: Complementary tool to node-exporter for monitoring more granular metrics
Usage: monitor-agent [-h] [-n namespace] [-i interval]

Options:
  -h    this help
  -i interval
        set interval for target monitor interval (default 60)
  -n namespace
        set namespace for target monitor namespace, default all namespace

./monitor-agent -i 30 -n hipster
```

### Running agent at each node with ansible
```
git clone http://gitlab.dds-sysu.tech/yu/monitor-agent.git /root/deploy
cd /root/deploy/monitor-agent

// start
ansible-playbook deploy/deploy.yml

curl http://33.33.33.223:9101/metrics
...
# HELP monitor_agent_io_read_syscall_total read syscalls:Attempt to count the number of read I/O operations--that is, system calls such as read(2) and pread(2).
# TYPE monitor_agent_io_read_syscall_total counter
monitor_agent_io_read_syscall_total{container="chaosblade-tool",namespace="kube-system",pod="chaosblade-tool-dm4kl"} 1.789723e+06
monitor_agent_io_read_syscall_total{container="kube-flannel",namespace="kube-system",pod="kube-flannel-ds-amd64-sll5k"} 1.1050566e+08
monitor_agent_io_read_syscall_total{container="node-cache",namespace="kube-system",pod="node-local-dns-pf4j5"} 8.4651229e+07
monitor_agent_io_read_syscall_total{container="node-exporter",namespace="istio-system",pod="node-exporter-r25l5"} 7.7026663e+07
monitor_agent_io_read_syscall_total{container="otel-agent",namespace="observability",pod="otel-agent-p2tbq"} 8.71326e+06
monitor_agent_io_read_syscall_total{container="promtail",namespace="observability",pod="promtail-52vpl"} 5.5649321e+07
# HELP monitor_agent_io_write_syscall_total write syscalls: count the number of writer I/O operations--that is, system calls such as write(2) and pwrite(2)
# TYPE monitor_agent_io_write_syscall_total counter
monitor_agent_io_write_syscall_total{container="chaosblade-tool",namespace="kube-system",pod="chaosblade-tool-dm4kl"} 5
monitor_agent_io_write_syscall_total{container="kube-flannel",namespace="kube-system",pod="kube-flannel-ds-amd64-sll5k"} 2.5592673e+07
monitor_agent_io_write_syscall_total{container="node-cache",namespace="kube-system",pod="node-local-dns-pf4j5"} 2.3058625e+07
monitor_agent_io_write_syscall_total{container="node-exporter",namespace="istio-system",pod="node-exporter-r25l5"} 7.477983e+06
monitor_agent_io_write_syscall_total{container="otel-agent",namespace="observability",pod="otel-agent-p2tbq"} 2.953108e+06
monitor_agent_io_write_syscall_total{container="promtail",namespace="observability",pod="promtail-52vpl"} 5.5673209e+07


// kill
ansible-playbook -u root deploy/stop.yml
```

### Reconfigure the promethues config
```
vim prometheus-config.yaml
    ...
    - job_name: 'monitor-agent'
      scrape_interval: 30s
      scrape_timeout: 10s
      metrics_path: /metrics
      static_configs:
        - targets: ['33.33.33.115:9101','33.33.33.116:9101','33.33.33.155:9101','33.33.33.167:9101','33.33.33.223:9101','33.33.33.227:9101','33.33.33.238:9101','33.33.33.60:9101','33.33.33.74:9101','33.33.33.76:9101','33.33.33.79:9101','33.33.33.80:9101','33.33.33.82:9101']

kubectl apply -f prometheus-config.yaml

```

### Restart Prometheus
```
export PROMETHEUS=$(kubectl get pod -n istio-system  | grep prometheus |awk '{print $1}' | head -n 1)
kubectl exec -it -n istio-system  $PROMETHEUS -c prometheus-server -- kill -HUP 1
```

### Result
![result](./figure/prometheus_query.jpg)