Skip to main content

​
Overview

The Sentinel service is a lightweight Go service that runs as its own single-replica Deployment in the same namespace as your Remote Execution Agent components (worker, Dag processor, triggerer). It provides real-time monitoring and operational visibility for agents running in your Kubernetes cluster. Sentinel provides:
  • Health aggregation: Every 30 seconds, Sentinel evaluates the health of all agent Pods in its namespace and sends a single aggregate heartbeat to the Astro control plane. The evaluation interval is set by reconcile_interval. The heartbeat cadence is set by the Astro control plane, which returns the interval to use in each heartbeat response.
  • Independent reporting path: Sentinel observes agents from outside the agent process, so it keeps reporting infrastructure-level status even when an agent component is unhealthy or restarting.
Sentinel is included in the Remote Execution Agent Helm chart starting in version 1.2.0. Astronomer recommends enabling Sentinel for all Remote Execution deployments.
Sentinel is namespace-scoped: each Sentinel instance only watches the Pods in its own namespace. If you run two agent deployments in separate namespaces, each has its own Sentinel instance.
The health data Sentinel reports is used internally by Astronomer Customer Support to monitor and triage agent issues. It isn’t exposed to you in the Astro UI or any customer-facing interface.

​
What Sentinel monitors

Sentinel evaluates agent health using these sources: If any Pod shows a non-Running or non-Ready state, or reports a backend connectivity failure through its /status endpoint, Sentinel reflects that in its aggregate heartbeat to the control plane. Pods that are still starting report an initializing state instead of an unhealthy one, so a normal rollout doesn’t register as a failure. The following table summarizes the failure scenarios that Sentinel detects:

​
Understand monitoring limitations

Sentinel monitors infrastructure-level health. It doesn’t have visibility into application-level connectivity failures that leave a Pod in a Running and Ready state. The following scenarios aren’t detected by Sentinel: In these cases, the affected Pod remains Running and Ready from a Kubernetes perspective, and the agent’s /status endpoint continues to report healthy XCom, secrets, and state store backends. Sentinel’s aggregate heartbeat shows a healthy status even though outbound connectivity to the control plane is broken.
If you observe agent connectivity issues where Pods appear healthy but tasks aren’t executing, check the agent Pod logs for repeated PoolTimeout or HTTP error entries. These failures require a Pod restart to resolve. See Remote Execution Agent failure and recovery scenarios for more details about heartbeat behavior during connectivity disruptions.

​
Set up Sentinel

Sentinel is enabled by default in Remote Execution Agent Helm chart version 2.0.0 and later. If you use an earlier chart version, enable it explicitly in your Helm chart configuration:
values.yaml
Astronomer recommends that you host the Sentinel image in your organization’s registry and update the image reference in your Helm chart configuration. To customize which agent Pods Sentinel observes, set agent_component_app_label in Sentinel’s configuration. By default, Sentinel only monitors Pods labeled app=astro-agent, but you can restrict or broaden this scope as needed.
The sentinel.config value replaces Sentinel’s entire generated configuration instead of merging with it. If you set it, you must also supply every value the chart would otherwise generate, including api_server.base_url, the API token, agent_auth, and health.port. Omitting them prevents Sentinel from starting or from authenticating to the control plane.

​
Verify Sentinel is running

After you enable Sentinel, confirm it’s running in your cluster:
1

Check the Sentinel Pod status

Run the following command to verify the Sentinel Pod is running in your agent namespace:
The output shows the Sentinel Pod in Running status.
2

Review Sentinel logs

Check the Sentinel logs to confirm it’s sending heartbeats and evaluating agent health:
Look for Component health evaluation completed, which Sentinel logs after each health evaluation, and Heartbeat sent successfully, which it logs after each heartbeat to the control plane. Both appear at the default info log level.If every backend check reports unknown, Sentinel can’t authenticate to the agent /status endpoint. Confirm that the agent and Sentinel share the same value for sentinelAuthSecret.

​
Security and scope

  • Sentinel only observes Pods with a specific label in its namespace.
  • All status data flows outbound to Astro. Sentinel doesn’t require inbound connectivity from outside your cluster, but it does need in-namespace network access to each agent Pod’s http port, and Kubernetes permissions to list and watch Pods and events in its namespace.
  • No Dag code, task logs, or business data leaves your environment.