Skip to content

CoreDNS-GSLB: Health Checks

The GSLB plugin supports several types of health checks for backends. Each type can be configured per backend in the YAML configuration file.

Additionally, the GSLB plugin automatically adapts the healthcheck interval for each DNS record based on recent resolution activity.

  • If a record is not resolved (queried) for a duration longer than resolution_idle_timeout, the healthcheck interval for its backends is multiplied by healthcheck_idle_multiplier (default: 10, configurable in the Corefile).
  • As soon as a DNS query is received for the record, the interval returns to its normal value (scrape_interval).
  • This mechanism reduces unnecessary healthcheck traffic for rarely used records, while keeping healthchecks frequent for active records.

Example:

  • scrape_interval: 10s, resolution_idle_timeout: 3600s, healthcheck_idle_multiplier: 10
  • If no DNS query is received for 1 hour, healthchecks run every 100s instead of every 10s.
  • When a query is received, healthchecks resume every 10s.

This feature helps optimize resource usage and backend load in large or dynamic environments.

Healthcheck Profiles

You can define reusable health check profiles at the top level of your YAML configuration using the healthcheck_profiles key. Each profile defines a health check type and its parameters. Backends can then reference these profiles by name in their healthchecks list, instead of repeating the same configuration.

Example: 

healthcheck_profiles:
  https_default:
    type: http
    params:
      enable_tls: true
      port: 443
      uri: /
      expected_code: 200
      timeout: 5s

records:
  webapp.example.com.:
    backends:
      - address: 10.0.0.1
        healthchecks: [ https_default ]  # Reference the profile by name
        priority: 1

You can still use inline healthcheck definitions as before, or mix both approaches. If a backend's healthchecks list contains a string, it is interpreted as a profile name.

Global Healthcheck Profiles

You can define healthcheck profiles globally for all zones using the Corefile directive:

gslb {
    ...
    healthcheck_profiles healthcheck_profiles.yml
}

The file healthcheck_profiles.yml should contain:

healthcheck_profiles:
  https_default:
    type: http
    params:
      enable_tls: true
      port: 443
      uri: /
      expected_code: 200
      timeout: 5s
  • Global profiles are available to all YAML zone files.
  • If a profile with the same name exists locally in a zone YAML, the local one takes precedence.
  • You can reference a profile by name in any backend's healthchecks list.

Healthcheck Types

HTTP(S)

Checks the health of an HTTP or HTTPS endpoint by making a request and validating the response code and/or body.

The HTTP health check connects to backend.address on given params.port. The Host header is set based on params.host, which does not overwrite the target address. HTTP forwards are not followed.

mTLS is enabled by setting params.enable_tls to true and then supplying both of params.cert_path and params.key_path there is the optional params.ca_path as well for using private PKI, this can be the Root CA or the chain depending on your server implementation of certificate checking strictness. If the CA path is omitted, the trusted root store is used.

If params.enable_tls is not enabled, mTLS will not be attempted, be explicit in your configuration intention! Also, if params.ca_path is set but params.skip_tls_verify is set to true, it will have a net-zero effect.

Additionally, the params.host is used as the SNI parameter in the Client Hello for multiplexed server targets.

Debugging HTTP health checks is easier when the user agent is clearly visible in the server logs, so a predefined User-Agent value is set to CoreDNS/GSLB/....

healthchecks:
  - type: http
    params:
      port: 443                # Port to connect (443 for HTTPS, 80 for HTTP)
      uri: "/"                 # URI to request
      method: "GET"            # HTTP method
      host: "localhost"        # Host header for the request
      headers:                 # Additional HTTP headers (key-value pairs)
      timeout: 5s              # Timeout for the HTTP request
      expected_code: 200       # Expected HTTP status code
      expected_body: ""        # Expected response body (empty means no body validation)
      enable_tls: true         # Use TLS for the health check (HTTPS)
      cert_path: "cert.pem"    # Define path to client certificate for mTLS
      key_path: "key.pem"      # Define path for client private key for mTLS
      ca_path: "cacert.pem"    # Define optional path to trusted Private RootCA File or Chain
      skip_tls_verify: true    # Skip TLS certificate validation

TCP

Checks if a TCP connection can be established to the backend on a given port.

healthchecks:
  - type: tcp
    params:
      port: 80         # TCP port to connect to
      timeout: "3s"    # Connection timeout

ICMP (Ping)

Checks if the backend responds to ICMP echo requests (ping).

healthchecks:
  - type: icmp
    params:
      timeout: 2s   # Timeout for the ICMP request
      count:  3     # Number of ICMP requests to send

MySQL

Checks MySQL server health by connecting and executing a query.

healthchecks:
  - type: mysql
    params:
      host: "10.0.0.5"         # MySQL server address
      port: 3306               # MySQL port
      user: "gslbcheck"        # Username
      password: "secret"       # Password
      database: "test"         # Database to connect
      timeout: "3s"            # Connection/query timeout
      query: "SELECT 1"        # Query to execute (optional, default: SELECT 1)

gRPC

Checks the health of a gRPC service using the standard gRPC health checking protocol (grpc.health.v1.Health/Check). It supports secure connections via TLS and mutual TLS (mTLS) client verification.

healthchecks:
  - type: grpc
    params:
      port: 9090                # gRPC port to connect to
      service: "grpc.health.v1.Health" # Service name (default: "")
      timeout: 5s               # Timeout for the gRPC request
      enable_tls: true          # Enable TLS/mTLS (default: false)
      cert_path: "cert.pem"     # Path to client certificate for mTLS (optional)
      key_path: "key.pem"       # Path to client private key for mTLS (optional)
      ca_path: "cacert.pem"     # Path to trusted CA Root certificate (optional)
      skip_tls_verify: false    # Skip TLS certificate validation (default: false)
  • service can be left empty to check the overall server health, or set to a specific service name.
  • enable_tls must be set to true to enable encrypted TLS transport.
  • cert_path and key_path are required when client certificate verification (mTLS) is enforced by the gRPC server.
  • ca_path allows specifying a private or custom CA root chain rather than the system's default trust store.

Lua Scripting

Executes an embedded Lua script to determine the backend health. The script can use the helper functions http_get(url) and json_decode(str) to perform HTTP requests and parse JSON. The global variable 'backend' provides the backend's address and priority.

Available helpers:

  • http_get(url, [timeout_sec], [user], [password], [tls_verify]): Performs an HTTP(S) GET request. Optional timeout (seconds), HTTP Basic auth (user, password), and TLS verification (default true).
  • json_decode(str): Parses a JSON string and returns a Lua table (or nil on error).
  • metric_get(url, metric_name, [timeout_sec], [tls_verify], [user], [password]): Fetches the value of a Prometheus metric from a /metrics endpoint (returns the first value found as a number or string, or nil if not found). Optional timeout (seconds), TLS verification (default true), and HTTP Basic auth (user, password).
  • ssh_exec(host, user, password_or_key_path, command, [timeout_sec], [key_passphrase]): Executes a command via SSH and returns the output as a string. If the third parameter points to a valid file path, it is treated as the path to an SSH private key file. Optional timeout (seconds) and key passphrase.
  • backend: A Lua table with fields:
    • address: the backend's address (string)
    • priority: the backend's priority (number)

Example: Use http_get and json_decode 

healthchecks:
  - type: lua
    params:
      timeout: 5s
      script: |
        local health = json_decode(http_get("http://" .. backend.address .. ":9200/_cluster/health"))
        if health and health.status == "green" and health.number_of_nodes >= 3 then
          return true
        else
          return false
        end

Example: Get a Prometheus metric value 

healthchecks:
  - type: lua
    params:
      timeout: 5s
      script: |
        local value = metric_get("http://myapp:9100/metrics", "nginx_connections_active")
        if value and value < 100 then
          return true
        end
        return false

Example: Check a process via SSH 

healthchecks:
  - type: lua
    params:
      timeout: 5s
      script: |
        local output = ssh_exec("10.0.0.5", "monitor", "secret", "pgrep nginx")
        if output and output ~= "" then
          return true
        else
          return false
        end

Example: metric_get with timeout and skip TLS verification 

healthchecks:
  - type: lua
    params:
      timeout: 5s
      script: |
        local value = metric_get("https://myapp:9100/metrics", "nginx_connections_active", 2, false)
        if value and value < 100 then
          return true
        end
        return false

Example: ssh_exec with timeout 

healthchecks:
  - type: lua
    params:
      timeout: 5s
      script: |
        local out = ssh_exec("10.0.0.5", "user", "pass", "pgrep nginx", 3)
        if out ~= "" then
          return true
        end
        return false

Example: ssh_exec with SSH private key authentication 

healthchecks:
  - type: lua
    params:
      timeout: 5s
      script: |
        -- Pass the path to the private key file instead of a password.
        -- Optional: pass the key's passphrase as the 6th argument.
        local out = ssh_exec("10.0.0.5", "monitor", "/home/coredns/.ssh/id_rsa", "pgrep nginx", 3, "key-passphrase-if-any")
        if out ~= "" then
          return true
        end
        return false

Security Warning

Storing plaintext passwords directly in the YAML configuration or passing them via the REST API represents a major credential exposure and remote code execution (RCE) risk.

  • Avoid hardcoding plaintext passwords inside ssh_exec.
  • Prefer using SSH private key files and restrict the SSH user permissions on the target server (e.g., using a restricted shell or setting a command="..." prefix in the destination's authorized_keys file).
  • Secure the configuration files and the API endpoints with strict file permissions and strong basic authentication.

Example: metric_get with HTTP Basic authentication 

healthchecks:
  - type: lua
    params:
      timeout: 5s
      script: |
        local value = metric_get("https://myapp:9100/metrics", "nginx_connections_active", 2, true, "user", "pass")
        if value and value < 100 then
          return true
        end
        return false

Rise and Fall Thresholds

To prevent healthcheck flapping due to transient network issues, you can configure consecutive success/failure thresholds to transition a backend's status.

  • fall (default: 3): The number of consecutive failed checks required to mark an active backend as offline (DOWN).
  • rise (default: 2): The number of consecutive successful checks required to mark an offline backend as online (UP).

These thresholds can be defined globally in a healthcheck profile, or overridden on a per-backend basis.

graph LR
    subgraph UP_State ["State: UP (Online)"]
        A[Healthy] -->|"Failed check"| B(Increment failures)
        B -->|"Failures < fall"| A
        A -->|"Successful check"| A_Reset(Reset failure counter)
    end

    subgraph DOWN_State ["State: DOWN (Offline)"]
        C[Unhealthy] -->|"Successful check"| D(Increment successes)
        D -->|"Successes < rise"| C
        C -->|"Failed check"| C_Reset(Reset success counter)
    end

    B -->|"Failures == fall (default: 3)"| C
    D -->|"Successes == rise (default: 2)"| A

    style A fill:#dcfce7,stroke:#16a34a,stroke-width:1px,color:#14532d
    style C fill:#fee2e2,stroke:#dc2626,stroke-width:1px,color:#7f1d1d

Thresholds Configured via Profiles example:

healthcheck_profiles:
  http_flapping_prevented:
    type: http
    rise: 4
    fall: 5
    params:
      port: 8080
      uri: /health

Thresholds Overridden Per-Backend example:

records:
  webapp.example.com.:
    backends:
      - address: "10.0.0.5"
        healthchecks: [ http_flapping_prevented ]
        rise: 2 # overrides the profile's rise value
        fall: 3 # overrides the profile's fall value

Healthcheck Bypass

In multi-region setups, you might want to bypass health checking for remote backends (e.g. to prevent WAN/cross-region network blips from marking them as down in a local resolver). You can specify assume_healthy: true on any backend:

backends:
  - address: "172.16.0.10"
    healthchecks: [ https_default ] # Checked normally
  - address: "172.16.0.20"
    assume_healthy: true            # No health probes executed, always assumed healthy
  • If assume_healthy: true is set, no health checks are performed for this backend, and IsHealthy() always returns true (as long as enable is also true).
  • It defaults to false.