Read-only root filesystem

In Astro Private Cloud (APC), the root filesystem of every platform container and every Airflow Deployment container is read-only. Kubernetes enforces this by setting readOnlyRootFilesystem: true on each container’s securityContext which reduces the container attack surface. This document explains the behavior, lists the locations that are writable by default, and shows how to add additional writable directories when an application needs to write outside of those paths.

Default behavior

Every container in the APC platform Helm chart, the Airflow Helm chart, and the Airflow operator runs with readOnlyRootFilesystem: true.
For Airflow Deployment Pods, /usr/local/airflow and /tmp are mounted as emptyDir volumes so that Airflow, your Dags, and provider packages can continue to write to these standard locations. An init container, usr-local-airflow-copier, copies the image’s /usr/local/airflow contents into that writable volume before the main container starts.
Other paths inside the container image, such as /, /etc, /var, and /home, are read-only. Any process that tries to create or modify a file outside of a mounted writable volume fails with a Read-only file system error.

If your application or Dag needs to write to disk, write to /usr/local/airflow/logs if the data is log output, or to a directory you have explicitly mounted with extraVolumes and extraVolumeMounts. Writing to other paths under /usr/local/airflow fails because the root filesystem is read-only.

Disable read-only root filesystem hardening

Astro Private Cloud 2.0.3 introduced the ability to disable this hardening for Airflow Deployment containers, by setting deployments.securityContext.container.readOnlyRootFilesystem: false. This fix will be included in future APC 2.x releases, but isn’t present in every 2.x version — for example, 2.1.0 doesn’t include it. Check your version’s release notes to confirm it’s available before relying on this key. Where it’s available, this is a config governance key — set it as a platform Helm value (astronomer.houston.config.deployments.securityContext.container.readOnlyRootFilesystem) to change the default for every Deployment, or scope it to a specific cluster, Workspace, or Deployment through a configuration override. See Config governance. Setting this to false removes readOnlyRootFilesystem from the containers’ security contexts and removes the usr-local-airflow-copier init container and its writable volume entirely, since there’s no longer a read-only filesystem to work around.

Disabling this hardening removes a container-attack-surface protection, so scope it as narrowly as you can rather than platform-wide. Reasonable cases to consider it: a large /usr/local/airflow directory that’s slow enough to copy that it pushes Pod scheduling past task SLAs under load, or an application or Dag that writes to a path under the read-only filesystem that you can’t redirect elsewhere with extraVolumes and extraVolumeMounts (see the sections above) or an environment variable the application respects. If one of those narrower fixes solves your specific problem, prefer it over disabling the hardening entirely.

Add a writable directory with `extraVolumes` and `extraVolumeMounts`

When an application must write to a path that isn’t writable by default, mount an emptyDir (or other ephemeral volume) at that path using the extraVolumes and extraVolumeMounts keys on the relevant component. The Airflow Helm chart exposes extraVolumes and extraVolumeMounts on each Airflow component, including:

airflow.scheduler
airflow.workers
airflow.triggerer
airflow.dagProcessor
airflow.apiServer
airflow.webserver
airflow.pgbouncer
airflow.redis
airflow.statsd
airflow.flower
dagDeploy

The following example mounts an emptyDir at /opt/cache on Airflow workers so a provider package can write its cache:

airflow:
  workers:
    extraVolumes:
      - name: provider-cache
        emptyDir: {}
    extraVolumeMounts:
      - name: provider-cache
        mountPath: /opt/cache

Apply the same pattern to other components by setting extraVolumes and extraVolumeMounts under the appropriate component key.

Size the volume

By default, emptyDir volumes can grow until they exhaust the node’s ephemeral storage. Set sizeLimit to bound the volume:

airflow:
  workers:
    extraVolumes:
      - name: provider-cache
        emptyDir:
          sizeLimit: 1Gi
    extraVolumeMounts:
      - name: provider-cache
        mountPath: /opt/cache

For more guidance on sizing ephemeral storage, see Ephemeral storage configuration.

Seed a writable directory with image contents

extraVolumes mounts an empty directory at the target path, which hides any files that were baked into the container image at that location. If your application reads files from a path it also needs to write to, use an init container to copy the image contents into the writable volume before the main container starts. The following example makes /opt/app writable on Airflow workers while preserving files from the image:

airflow:
  workers:
    extraVolumes:
      - name: app-data
        emptyDir: {}
    extraVolumeMounts:
      - name: app-data
        mountPath: /opt/app
    extraInitContainers:
      - name: seed-app-data
        image: "{{ .Values.images.airflow.repository }}:{{ .Values.images.airflow.tag }}"
        command:
          - sh
          - -c
          - cp -a /opt/app/. /seed/
        volumeMounts:
          - name: app-data
            mountPath: /seed
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false

The init container shares the emptyDir volume with the main container. When the main container starts, the volume already contains the files from the image, and the main container can also write to it.

Redirect application write paths with environment variables

Some applications can be configured to write to a different location through environment variables. When that option exists, mount a writable volume with extraVolumes and extraVolumeMounts as shown earlier, and then point the application at that mount path. For example, mount an emptyDir at /tmp on Airflow workers and set TMPDIR so libraries that respect it write there:

airflow:
  workers:
    extraVolumes:
      - name: tmp
        emptyDir: {}
    extraVolumeMounts:
      - name: tmp
        mountPath: /tmp
    env:
      - name: TMPDIR
        value: /tmp

Check each application’s documentation for the variables it supports, such as TMPDIR, HOME, XDG_CACHE_HOME, or vendor-specific cache directory variables. Combining a single writable volume with an environment variable is often cheaper than mounting volumes at each path the application touches.

Troubleshoot read-only filesystem errors

If a Pod or task fails with an error such as Read-only file system, Permission denied, or Errno 30, the application is attempting to write to a path that isn’t backed by a writable volume.

Identify the path the application is writing to from the error message or task logs.
If the path should already be writable, such as /usr/local/airflow/logs or a directory you mounted with extraVolumes, verify that the Pod has the expected volumes with kubectl describe pod <pod-name>.
If the path isn’t writable, either:
- Add extraVolumes and extraVolumeMounts for the component that runs the application, as shown in the previous sections.
- Configure the application with an environment variable to write to a path you have mounted as writable.
Apply the change with a Helm upgrade and re-run the failing task.

Overview

Architecture

Upgrade Astro Private Cloud

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Read-only root filesystem

Default behavior

Disable read-only root filesystem hardening

Add a writable directory with `extraVolumes` and `extraVolumeMounts`

Size the volume

Seed a writable directory with image contents

Redirect application write paths with environment variables

Troubleshoot read-only filesystem errors

​Default behavior

​Disable read-only root filesystem hardening

​Add a writable directory with extraVolumes and extraVolumeMounts

​Size the volume

​Seed a writable directory with image contents

​Redirect application write paths with environment variables

​Troubleshoot read-only filesystem errors

Default behavior

Disable read-only root filesystem hardening

Add a writable directory with `extraVolumes` and `extraVolumeMounts`

Size the volume

Seed a writable directory with image contents

Redirect application write paths with environment variables

Troubleshoot read-only filesystem errors