Reference

Components

mimir

mimir.alerts.kubernetes

Experimental Open source

`mimir.alerts.kubernetes`

EXPERIMENTAL: This is an experimental component. Experimental components are subject to frequent breaking changes, and may be removed with no equivalent replacement. To enable and use an experimental component, you must set the stability.level flag to experimental.

mimir.alerts.kubernetes discovers AlertmanagerConfig Kubernetes resources and loads them into a Mimir instance.

You can specify multiple mimir.alerts.kubernetes components by giving them different labels.
Kubernetes label selectors can be used to limit the Namespace and AlertmanagerConfig resources considered during reconciliation.
Compatible with the Alertmanager APIs of Grafana Mimir, Grafana Cloud, and Grafana Enterprise Metrics.
Compatible with the AlertmanagerConfig CRD from the prometheus-operator.
This component accesses the Kubernetes REST API from within a Pod.

Note
This component requires Role-based access control (RBAC) to be set up in Kubernetes in order for Alloy to access it via the Kubernetes REST API.

mimir.alerts.kubernetes does not support [clustering][clustered mode]. [clustered mode]: ../../../../get-started/clustering/

Usage

mimir.alerts.kubernetes "<LABEL>" {
  address = "<MIMIR_URL>"
}

Arguments

You can use the following arguments with mimir.alerts.kubernetes:

Name	Type	Description	Default	Required
`address`	`string`	URL of the Mimir Alertmanager.		yes
`global_config`	`string`	Alertmanager configuration to be merged with AlertmanagerConfig CRDs.		yes
`bearer_token_file`	`string`	File containing a bearer token to authenticate with.		no
`bearer_token`	`secret`	Bearer token to authenticate with.		no
`enable_http2`	`bool`	Whether HTTP2 is supported for requests.	`true`	no
`follow_redirects`	`bool`	Whether redirects returned by the server should be followed.	`true`	no
`http_headers`	`map(list(secret))`	Custom HTTP headers to be sent along with each request. The map key is the header name.		no
`no_proxy`	`string`	Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying.		no
`proxy_connect_header`	`map(list(secret))`	Specifies headers to send to proxies during CONNECT requests.		no
`proxy_from_environment`	`bool`	Use the proxy URL indicated by environment variables.	`false`	no
`proxy_url`	`string`	HTTP proxy to send requests through.		no
`template_files`	`map(string)`	A map of Alertmanager template files.	`{}`	no

At most, one of the following can be provided:

authorization block
basic_auth block
bearer_token_fileargument
bearer_token argument
oauth2 block

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_from_environment uses the environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO_PROXY. proxy_url and no_proxy must not be configured if proxy_from_environment is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

Blocks

The following blocks are supported inside the definition of mimir.alerts.kubernetes:

Block	Description	Required
`authorization`	Configure generic authorization to the endpoint.	no
`basic_auth`	Configure `basic_auth` for authenticating to the endpoint.	no
`oauth2`	Configure OAuth 2.0 for authenticating to the endpoint.	no
`oauth2` > `tls_config`	Configure TLS settings for connecting to the endpoint.	no
`alertmanagerconfig_namespace_selector`	Label selector for `Namespace` resources.	no
`alertmanagerconfig_namespace_selector` > `match_expression`	Label match expression for `Namespace` resources.	no
`alertmanagerconfig_selector`	Label selector for `AlertmanagerConfig` resources.	no
`alertmanagerconfig_selector` > `match_expression`	Label match expression for `AlertmanagerConfig` resources.	no
`tls_config`	Configure TLS settings for connecting to the endpoint.	no

The > symbol indicates deeper levels of nesting. For example, oauth2 > tls_config refers to a tls_config block defined inside an oauth2 block.

`authorization`

Name	Type	Description	Required
`credentials_file`	`string`	File containing the secret value.	no
`credentials`	`secret`	Secret value.	no
`type`	`string`	Authorization type, for example, “Bearer”.	no

credential and credentials_file are mutually exclusive, and only one can be provided inside an authorization block.

`basic_auth`

Name	Type	Description	Required
`password_file`	`string`	File containing the basic auth password.	no
`password`	`secret`	Basic auth password.	no
`username`	`string`	Basic auth username.	no

password and password_file are mutually exclusive, and only one can be provided inside a basic_auth block.

`alertmanagerconfig_selector` and `alertmanagerconfig_namespace_selector`

The alertmanagerconfig_selector and alertmanagerconfig_namespace_selector blocks describe a Kubernetes label selector for AlertmanagerConfig CRDs or namespace discovery.

The following arguments are supported:

Name	Type	Description	Default	Required
`match_labels`	`map(string)`	Label keys and values used to discover resources.	`{}`	yes

When the match_labels argument is empty, all resources are matched.

`match_expression`

The match_expression block describes a Kubernetes label match expression for AlertmanagerConfig CRDs or namespace discovery.

The following arguments are supported:

Name	Type	Description	Required
`key`	`string`	The label name to match against.	yes
`operator`	`string`	The operator to use when matching.	yes
`values`	`list(string)`	The values used when matching.	no

The operator argument should be one of the following strings:

"In"
"NotIn"
"Exists"
"DoesNotExist"

The values argument must not be provided when operator is set to "Exists" or "DoesNotExist".

`oauth2`

Name	Type	Description	Default	Required
`client_id`	`string`	OAuth2 client ID.		no
`client_secret_file`	`string`	File containing the OAuth2 client secret.		no
`client_secret`	`secret`	OAuth2 client secret.		no
`endpoint_params`	`map(string)`	Optional parameters to append to the token URL.		no
`no_proxy`	`string`	Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying.		no
`proxy_connect_header`	`map(list(secret))`	Specifies headers to send to proxies during CONNECT requests.		no
`proxy_from_environment`	`bool`	Use the proxy URL indicated by environment variables.	`false`	no
`proxy_url`	`string`	HTTP proxy to send requests through.		no
`scopes`	`list(string)`	List of scopes to authenticate with.		no
`token_url`	`string`	URL to fetch the token from.		no

client_secret and client_secret_file are mutually exclusive, and only one can be provided inside an oauth2 block.

The oauth2 block may also contain a separate tls_config sub-block.

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

`tls_config`

Name	Type	Description	Required
`ca_pem`	`string`	CA PEM-encoded text to validate the server with.	no
`ca_file`	`string`	CA certificate to validate the server with.	no
`cert_pem`	`string`	Certificate PEM-encoded text for client authentication.	no
`cert_file`	`string`	Certificate file for client authentication.	no
`insecure_skip_verify`	`bool`	Disables validation of the server certificate.	no
`key_file`	`string`	Key file for client authentication.	no
`key_pem`	`secret`	Key PEM-encoded text for client authentication.	no
`min_version`	`string`	Minimum acceptable TLS version.	no
`server_name`	`string`	ServerName extension to indicate the name of the server.	no

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

ca_pem and ca_file
cert_pem and cert_file
key_pem and key_file

When configuring client authentication, both the client certificate (using cert_pem or cert_file) and the client key (using key_pem or key_file) must be provided.

When min_version isn’t provided, the minimum acceptable TLS version is inherited from Go’s default minimum version, TLS 1.2. If min_version is provided, it must be set to one of the following strings:

"TLS10" (TLS 1.0)
"TLS11" (TLS 1.1)
"TLS12" (TLS 1.2)
"TLS13" (TLS 1.3)

Exported fields

mimir.alerts.kubernetes doesn’t export any fields.

Component health

mimir.alerts.kubernetes is reported as unhealthy if given an invalid configuration or an error occurs during reconciliation.

Debug information

mimir.alerts.kubernetes does not expose debug information.

Debug metrics

Metric Name	Type	Description
`mimir_alerts_client_request_duration_seconds`	`histogram`	Duration of requests to the Mimir API.
`mimir_alerts_config_updates_total`	`counter`	Number of times the configuration has been updated.
`mimir_alerts_events_failed_total`	`counter`	Number of events that failed to be processed, partitioned by event type.
`mimir_alerts_events_retried_total`	`counter`	Number of events that were retried, partitioned by event type.
`mimir_alerts_events_total`	`counter`	Number of events processed, partitioned by event type.

Example

This example creates a mimir.alerts.kubernetes component which only loads namespace and AlertmanagerConfig resources if they contain an alloy label set to yes.

remote.kubernetes.configmap "default" {
  namespace = "default"
  name      = "alertmgr-global"
}

mimir.alerts.kubernetes "default" {
  address       = "http://mimir-nginx.mimir-test.svc:80"
  global_config = remote.kubernetes.configmap.default.data["glbl"]

  template_files = {
    `default_template` = 
`{{ define "__alertmanager" }}AlertManager{{ end }}
{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}`,
  }

  alertmanagerconfig_selector {
      match_labels = {
          alloy = "yes",
      }
  }

  alertmanagerconfig_namespace_selector {
      match_labels = {
          alloy = "yes",
      }
  }
}

The following example is an RBAC configuration for Kubernetes. It authorizes Alloy to query the Kubernetes REST API:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: alloy
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: alloy
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["alertmanagerconfigs"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: alloy
subjects:
- kind: ServiceAccount
  name: alloy
  namespace: default
roleRef:
  kind: ClusterRole
  name: alloy
  apiGroup: rbac.authorization.k8s.io

The following is an example of a complete Kubernetes configuration:

apiVersion: v1
kind: Namespace
metadata:
  name: testing
  labels:
    alloy: "yes"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana-alloy
  namespace: testing
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-alloy
rules:
- apiGroups: [""]
  resources: ["namespaces", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["alertmanagerconfigs"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: grafana-alloy
subjects:
- kind: ServiceAccount
  name: grafana-alloy
  namespace: testing
roleRef:
  kind: ClusterRole
  name: grafana-alloy
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-alloy
spec:
  type: NodePort
  selector:
    app: grafana-alloy
  ports:
      # By default and for convenience, the `targetPort` is set to the same value as the `port` field.
    - port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: testing
  name: grafana-alloy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana-alloy
  template:
    metadata:
      labels:
        app: grafana-alloy
    spec:
      serviceAccount: grafana-alloy
      containers:
      - name: alloy
        image: grafana/alloy:latest
        imagePullPolicy: Never
        args:
        - run
        - /etc/config/config.alloy
        - --stability.level=experimental
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
        - name: config-volume
          configMap:
            name: alloy-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alloy-config
  namespace: testing
data:
  config.alloy: |
      remote.kubernetes.configmap "default" {
        namespace = "testing"
        name = "alertmgr-global"
      }

      mimir.alerts.kubernetes "default" {
        address = "http://mimir-nginx.mimir-test.svc:80"
        global_config = remote.kubernetes.configmap.default.data["glbl"]
        template_files = {
          `default_template` = 
      `{{ define "__alertmanager" }}AlertManager{{ end }}
      {{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}`,
        }
        alertmanagerconfig_namespace_selector {
            match_labels = {
                alloy = "yes",
            }
        }
        alertmanagerconfig_selector {
            match_labels = {
                alloy = "yes",
            }
        }
      }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmgr-global
  namespace: testing
data:
  glbl: |
    global:
      resolve_timeout: 5m
      http_config:
        follow_redirects: true
        enable_http2: true
      smtp_hello: localhost
      smtp_require_tls: true
    route:
      receiver: "null"
    receivers:
    - name: "null"
    - name: "alloy-namespace/global-config/myreceiver"
    templates:
    - 'default_template'
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmgr-config1
  namespace: testing
  labels:
    alloy: "yes"
spec:
  route:
    receiver: "null"
    routes:
    - receiver: myamc
      continue: true
  receivers:
  - name: "null"
  - name: myamc
    webhookConfigs:
    - url: http://test.url
      httpConfig:
        followRedirects: true
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmgr-config2
  namespace: testing
  labels:
    alloy: "yes"
spec:
  route:
    receiver: "null"
    routes:
    - receiver: 'database-pager'
      group_wait: 10s
      matchers:
      - name: service
        value: webapp
  receivers:
  - name: "null"
  - name: "database-pager"

The Kubernetes configuration above will create the Alertmanager configuration below and send it to Mimir:

template_files:
    default_template: |-
        {{ define "__alertmanager" }}AlertManager{{ end }}
        {{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}
alertmanager_config: |
    global:
      resolve_timeout: 5m
      http_config:
        follow_redirects: true
        enable_http2: true
      smtp_hello: localhost
      smtp_require_tls: true
    route:
      receiver: "null"
      continue: false
    receivers:
    - name: "null"
    - name: alloy-namespace/global-config/myreceiver
    templates:
    - default_template

Was this page helpful?

Email docs@grafana.com

Help and support

Community

mimir.alerts.kubernetes

Usage

Arguments

Blocks

authorization

basic_auth

alertmanagerconfig_selector and alertmanagerconfig_namespace_selector

match_expression

oauth2

tls_config

Exported fields

Component health

Debug information

Debug metrics

Example

Was this page helpful?

`mimir.alerts.kubernetes`

`authorization`

`basic_auth`

`alertmanagerconfig_selector` and `alertmanagerconfig_namespace_selector`

`match_expression`

`oauth2`

`tls_config`