# autoheal **Repository Path**: mirrors_openshift/autoheal ## Basic Information - **Project Name**: autoheal - **Description**: Autoheals based on monitoring alerts - **Primary Language**: Unknown - **License**: Apache-2.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2020-08-22 - **Last Updated**: 2026-02-14 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Auto-heal Service This project contains the _auto-heal_ service. It receives alert notifications from the Prometheus alert manager and executes Ansible playbooks to resolve the root cause. ## Configuration Most of the configuration of the auto-heal service is kept in a YAML configuration file. The name of the configuration file is specified using the `--config-file` command line option. If this option isn't explicitly given then the service will try to load the `autoheal.yml` file from the current working directory. In addition to the configuration file the auto-heal service also uses command line options to configure the connection to the Kubernetes API and the log level. Use the `-h` option to get a complete list of these command line options. The `--kubeconfig` command line option is used to specify the location of the Kubernetes client configuration file. When running outside of a Kubernetes cluster the auto-heal service will use `$HOME/.kube/config` by default, the same used by the `kubectl` command. When running inside a Kubernetes cluster it will use the configuration that Kubernetes mounts automatically in the pod file system. So in most cases this command line option won't have to be explicitly included. Assuming that you want to have your own `my.yml` configuration file a typical command line will be the following: ```bash $ autoheal server --config-file=my.yml --logtostderr ``` See the `autoheal.yml` file for a complete example. ### AWX or AnsibleTower configuration The first section of the configuration file is named `awx` and it contains all the details needed to connect to the [AWX](https://www.ansible.com/products/awx-project) or [Ansible Tower](https://www.ansible.com/products/tower) server: ```yaml awx: address: https://myawx.example.com/api proxy: http://myproxy.example.com:3128 credentialsRef: namespace: my-namespace name: my-awx-credentials tlsRef: namespace: my-namespace name: my-awx-ca project: "Auto-heal" ``` The `address` parameter is the URL of the API of the AWX server. It should contain the `/api` suffix, but not the `/v1` or `/v2` suffix, as the auto-heal service will internally decide which version to use. The `proxy` parameter is optional, and it indicates what HTTP proxy should be used to connect to the AWX API. If this parameter is not specified, or if it is empty, then the connection will be direct to the AWX server, without a proxy. The `credentialsRef` parameter is a reference to the [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret) that contains the user name and password used to connect to the AWX API. That secret should contain the `username` and `password` keys. For example: ```yaml apiVersion: v1 kind: Secret metadata: namespace: my-namespace name: my-awx-credentials data: username: YWxlcnQtaGVhbGVy password: ... ``` Alternatively it is also possible to specify the user name and password directly inside the configuration file, using the `credentials` section. For example: ```yaml credentials: username: autoheal password: ... ``` This is very convenient for development environments, but it is not recommended for production environments, as then the configuration file needs to be protected very carefully. For example, you can create a separate file for the credentials, give it restricted permissions, and then load it using the `--config-file` option twice: ``` $ echo > general.yml <<. awx: address: https://myawx.example.com/api . $ echo > credentials.yml <<. credentials: username: "autoheal" password: "..." . $ chmod u=r,g=,o= credentials.yml $ autoheal server --config-file=general.yml --config-file=credentials.yml ``` The `tlsRef` parameter is a reference to the [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret) that contains the certificates used to connect to the AWX API. That secret should contain the `ca.crt` key, for example: ```yaml apiVersion: v1 kind: Secret metadata: namespace: my-namespace name: my-awx-tls data: ca.crt: |- LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvVENDQWVXZ0F3SUJBZ0lKQUxNRXB6OWxa VkVzdzI3Sm5BYlMyejNhbUF0YTc1QmNnVGcvOUFCdDV0VVc2VTJOKzkKbXc9PQotLS0tLUVORCBD ... ``` Alternatively it is also possible to specify the CA certificates directly inside the configuration file, using the `tls` section. For example: ```yaml tls: caCerts: |- -----BEGIN CERTIFICATE----- MIIFgzCCA2ugAwIBAgIPXZONMGc2yAYdGsdUhGkHMA0GCSqGSIb3DQEBCwUAMDsx CzAJBgNVBAYTAkVTMREwDwYDVQQKDAhGTk1ULVJDTTEZMBcGA1UECwwQQUMgUkFJ ... -----END CERTIFICATE----- ``` They can also be specified indirectly, putting the name of a PEM file in the `caFile` parameter: ```yaml tls: caFile: /etc/autoheal/my-ca.pem ``` The `insecure` parameter controls whether to use an insecure connection to the AWX server. If the connection is insecure then the TLS will not be verified. It should always be set to `false` (the default) in production environments. The `project` parameter is the name of the AWX project that contains the job templates that will be used to run the playbooks. ### Throttling configuration The `throttling` section of the configuration describes how to throttle the execution of healing actions. This is intended to prevent _healing storms_ that could happen if the same alerts are send repeatedly to the service. The `interval` parameter controls the time that the service will remember an executed healing action. If an action is triggered more than once in the given interval it will be executed only the first time. The rest of the times it will be logged and ignored. (see `autoheal.yml` for an example.) The default interval value is one hour. Leaving the `interval` parameter 0 will *disable* throttling altogether. Note that for throttling purposes actions are considered the same if they have exactly the same fields with exactly the same values *after* processing them as templates. For example, an action defined like this: ```yaml awxJob: template: "Restart {{ $labels.service }}" ``` Will have different values for the `template` field if the triggering alerts have different `service` labels. The auto-heal service performs a periodic job status check against AWX server, to check the status of the active jobs that were triggered. The `jobStatusCheckInterval` parameter determines how often to perform this check. It is optional, and the defult is '5m' (every 5 minutes). ### Healing rules configuration The second important section of the configuration file is `rules`. It contains the list of _healing rules_ used by the auto-heal service to decide which action to run for each received alert. For example: ```yaml rules: - metadata: name: start-node labels: alertname: "NodeDown" awxJob: template: "Start node" extraVars: node: "{{ $labels.instance }}" - metadata: name: start-service labels: alertname: ".*Down" service: ".*" awxJob: template: "Start service" ``` The above example contains two _healing rules_. The first rule will be executed when the alert received contains a label named `alertname` with a value that matches the regular expression `NodeDown`. The second rule will be executed when the alert received contains a labels `alertname` *and* `service`, matching the regular expressions `.*Down` and `.*` respectively. The `metadata` parameter of each rule is used to specify the `name` of the rule, which is used by the auto-heal service to reference it in log messages and in metrics. The `labels` and `annotations` parameters of a rule are maps of strings used to specify the labels and annotations that the alerts should contain in order to match the rule. The keys of these maps are the names of the labels or annotations. The values of these maps are regular expressions that the values of those labels or annotations should match. The `awxJob` parameter indicates which job template should be executed when an alert matches the rule. The `template` parameter is the name of the AWX job template. The `extraVars` parameter is optional, and if specified it is used to pass additional variables to the playbook, like with the `--extra-vars` option of the `ansible-playbook` command. Regardless to the `extraVars` setting, the content of the alert that triggered the AWX job will be passed to the playbook as part of `extraVars`, in a variable named `alert`. The `limit` parameter is optional, and if specified it is passed to AWX to constrain the list of hosts managed or affected by the playbook. Multiple patterns can be separated by colons (`:`). As with core Ansible, `a:b` means "in group a or b", `a:b:&c` means "in a or b but must be in c", and `a:!b` means "in a, and definitely not in b". > Note that in order to be able to use `extraVars` and `limit` > mechanisms the AWX job template should have the > _Prompt on lauch_ box checked, otherwise the variables passed > will be ignored. The values of all the parameters inside `awxJob` are processed as [Go templates](https://golang.org/pkg/text/template) before executing the job. These templates receive the details of the alert inside the `$labels` and `$annotations` variables. For example, to generate dynamically the name of the job templates to execute from the value of the `template` annotation of the alert: ```yaml awxJob: template: "{{ $annotations.template }}" ``` Or to pass a variable `node` to the playbook, calculated from the `instance` label: ```yaml awxJob: template: "My template" extraVars: node: "{{ $labels.node }}" ``` Limit execution to a host, calculated from the `instance` label: ```yaml awxJob: template: "My template" limit: "{{ $labels.instance }}" ``` ### Alertmanager Configuration Follow the upstream [Prometheus Alertmanager documentation](https://prometheus.io/docs/alerting/configuration/) to configure alerts. For reference, here is an example Alertmanager configuration that sends an alert to the auto-heal service with authentication. This example assumes autoheal and the Alertmanager are running on the same OpenShift cluster, and requires Alertmanager 0.15 or newer. ```yaml global: resolve_timeout: 1m route: group_wait: 1s group_interval: 1s repeat_interval: 5m receiver: autoheal routes: - match: alertname: DeadMansSwitch repeat_interval: 5m receiver: autoheal receivers: - name: default - name: deadmansswitch - name: autoheal webhook_configs: - url: https://autoheal.openshift-autoheal.svc/alerts http_config: bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ``` When using the cluster-monitoring-operator, save the configuration as `alertmanager.yaml` and use this command to apply it: ```oc create secret generic alertmanager-main \ --namespace=openshift-monitoring \ --from-literal=alertmanager.yaml="$(< alertmanager.yaml)" \ --dry-run -oyaml \ | \ oc replace secret \ --namespace=openshift-monitoring \ --filename=- ``` ## Building To build the binary run this command: ``` $ make ``` To build the RPM and the images, run this command: ``` $ make build-images ``` ## Testing To run the automated tests of the project run this command: ``` $ make check ``` To manually test the service, without having to have a running Prometheus alert manager that generates the alert notifications, you can use the `*-alert.json` files that are inside the `examples` directory. For example, to simulate the `NodeDown` alert start the server and then use [curl](https://curl.haxx.se) to send the alert notification: ``` $ autoheal server --config-file=my.yml $ curl --data @examples/node-down-alert.json http://localhost:9099/alerts ``` ## Installing To install the service to an _OpenShift_ cluster use the template contained in the `template.yml` file. This template requires at the very minimum the address and the credentials to connect to the AWX or Ansible Tower server. See the `template.sh` script for an example of how to use it. ## Development If needed for development, we can run the server without an OpenShift cluster, simulating OpenShift's alert manager using curl commands. In the examples dir we have examples of firing alerts, and a configuration file that does not require a connection to a working OpenShift cluster. To run autoheal in dev mode (without a running OpenShift cluster) developers can use the dev config file in the examples dir. To simulate alerts firing, developers can use the example alerts. ``` $ make build $ make run-dev ``` ``` $ curl --data @examples/node-down-alert.json http://localhost:9099/alerts ``` When developing features that does not require AWX server, developers can use a mock-awx server from the examples dir. The mock server will listen on port 8080. ``` $ cd examples/mock-awx $ go run mock-awx.go ```