Deploy DAGs with NFS

You can use an external Network File System (NFS) Volume to deploy Dags to an Airflow Deployment on Astro Private Cloud (APC). Unlike deploying Dags with the Astro CLI, deploying Dags to an NFS volume doesn’t require rebuilding a Docker image and restarting your underlying Airflow service. When a Dag is added to an NFS volume, it automatically appears in the Airflow UI without requiring additional action or causing downtime.

How NFS deploys work

When you configure an NFS volume for a Deployment:

APC validates the NFS location format (SERVER_IP:PATH).
APC creates a Kubernetes PersistentVolume (PV) pointing to your NFS server.
APC creates a PersistentVolumeClaim (PVC) bound to the PV.
The NFS volume is mounted read-only to the scheduler and workers at /usr/local/airflow/dags.
Dags are synced by writing files directly to your NFS server.

NFS Server (/dags)
       │
       ▼
┌──────────────┐
│ Kubernetes   │
│ PV + PVC     │
└──────┬───────┘
       │
       ├──► Scheduler (/usr/local/airflow/dags)
       │
       └──► Workers (/usr/local/airflow/dags)

Implementation considerations

If you configure NFS for a Deployment, you can’t use the Astro CLI or service accounts to deploy DAGs to that Deployment. NFS becomes the exclusive deployment mechanism.

Before configuring NFS deploys:

Namespace pools limitation: NFS deploys will not work if you use namespace pools and set global.clusterRoles to false. The NFS deploy feature requires creating PersistentVolumes, which are cluster-scoped resources.
Dags only: NFS volumes deploy only Dags. To add Python dependencies or system packages, update your requirements.txt and packages.txt files and deploy using the CLI or CI/CD.
Airflow version: NFS volumes require Airflow 2.0 or later.
Read-only mount: The NFS volume is mounted read-only to Airflow components. Write operations must happen directly on the NFS server.

Prerequisites

APC 1.0 or later installed
An NFS server accessible from your Kubernetes cluster
Network connectivity between cluster nodes and the NFS server
Read access configured for UID/GID 50000 on the NFS share

Enable NFS volume storage

A System Admin must enable NFS deploys on the platform. Update your values.yaml:

houston:
  config:
    deployments:
      configureDagDeployment: true
      nfsMountDagDeployment: true

Apply the configuration change:

helm upgrade astronomer astronomer/astronomer \
  -f values.yaml \
  --namespace astronomer

Provision an NFS volume

AWS (EFS)
GCP (Filestore)
Azure (Azure Files)
Self-managed

Create an EFS file system.
Configure security groups to allow NFS traffic (port 2049) from your EKS nodes.
Create an access point or use the root directory.
Note the file system DNS name: fs-xxxxxxxx.efs.region.amazonaws.com.

Create a Premium File Storage account.
Create an NFS file share.
Configure network access from your AKS cluster.
Note the mount address: <storage-account>.file.core.windows.net:/<storage-account>/<share-name>.

For on-premises or self-managed NFS servers:

Ensure the NFS server exports a directory with appropriate permissions.
Configure /etc/exports to allow access from Kubernetes node IPs.
Configure /etc/exports to map client access to UID/GID 50000 using all_squash (see the example below and Security considerations).

Example /etc/exports:

/srv/airflow-dags 10.0.0.0/8(rw,sync,no_subtree_check,all_squash,anonuid=50000,anongid=50000)

Configure NFS for a Deployment

In the APC UI, create a new Deployment or open an existing one.
Go to DAG Deployment in the Deployment settings.
Select NFS Volume Mount as the mechanism.
Enter the NFS location in IP:PATH format:
- AWS EFS: 10.0.0.1:/
- GCP Filestore: 10.0.0.1:/dags
- Azure Files: storage-account.file.core.windows.net:/storage-account/share-name
Click Save or Deploy Changes.

Create a new Deployment with NFS:

astro deployment create my-deployment \
  --nfs-location "192.168.0.1:/dags"

Update an existing Deployment to use NFS:

astro deployment update <deployment-id> \
  --nfs-location "192.168.0.1:/dags"

mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    label: "my-deployment"
    dagDeployment: {
      type: volume
      nfsLocation: "192.168.0.1:/dags"
    }
  ) {
    id
    releaseName
  }
}

Deploy Dags to NFS volume

Once configured, deploy Dags by copying files to your NFS server. The method depends on your infrastructure:

Direct copy

# Copy DAGs to NFS mount point
cp -r dags/* /mnt/nfs/dags/

Using kubectl

If you have a pod with NFS access:

kubectl cp dags/ deployment-namespace/nfs-sync-pod:/dags/

CI/CD integration

Example GitHub Actions workflow:

name: Deploy DAGs to NFS
on:
  push:
    branches: [main]
    paths: ['dags/**']

jobs:
  deploy:
    runs-on: self-hosted  # Runner with NFS access
    steps:
      - uses: actions/checkout@v4
      - name: Sync DAGs
        run: |
          rsync -av --delete dags/ /mnt/nfs/airflow-dags/

Sync from cloud storage

For cloud-native workflows, sync from object storage:

# AWS S3 to EFS
aws s3 sync s3://my-bucket/dags/ /mnt/efs/dags/

# GCS to Filestore  
gsutil -m rsync -r gs://my-bucket/dags/ /mnt/filestore/dags/

# Azure Blob to Azure Files
azcopy sync "https://account.blob.core.windows.net/dags" "/mnt/azure/dags"

Verify NFS configuration

Check that the PV and PVC were created:

# List PersistentVolumes
kubectl get pv | grep dags

# List PersistentVolumeClaims in the deployment namespace
kubectl get pvc -n <deployment-namespace> | grep dags

Verify the volume is mounted in Airflow pods:

kubectl exec -n <deployment-namespace> <scheduler-pod> -- \
  ls -la /usr/local/airflow/dags

Troubleshooting

Dags aren’t appearing

Check NFS connectivity:

kubectl exec -n <namespace> <pod> -- \
  showmount -e <nfs-server-ip>

Verify mount permissions:

kubectl exec -n <namespace> <scheduler-pod> -- \
  ls -la /usr/local/airflow/dags

Check PV/PVC status:

kubectl describe pv <deployment>-dags-<hash>
kubectl describe pvc -n <namespace> <deployment>-dags-<hash>

Permission denied errors

Ensure your NFS export allows access for UID/GID 50000:

# On NFS server
chown -R 50000:50000 /srv/airflow-dags
chmod -R 755 /srv/airflow-dags

Stale file handle

If pods report stale NFS handles after server restart:

# Restart affected pods
kubectl rollout restart deployment -n <namespace> <scheduler>
kubectl rollout restart statefulset -n <namespace> <workers>

Network connectivity issues

Verify NFS port (2049) is accessible:

kubectl run nfs-test --rm -it --image=busybox -- \
  nc -zv <nfs-server-ip> 2049

Security considerations

APC mounts the NFS volume read-only into Airflow components, so a compromised Pod can’t modify Dags on the server. Beyond that, APC has no visibility into your NFS server configuration — most NFS security hardening is your responsibility.

In your Kubernetes cluster

Network policies: Restrict which Pods can reach the NFS server using Kubernetes network policies in the Deployment namespace.

On your NFS server

Export rules (/etc/exports): Restrict exports to specific Kubernetes node IPs or CIDR ranges, not broad networks. Avoid no_root_squash — it grants the client’s root user full root access on the NFS server, which is a significant security risk. Use root_squash or all_squash with anonuid=50000/anongid=50000 so Airflow can still read Dag files. The self-managed example above shows a working configuration.
Export path scoping: Export only the directory subtree used for Dags. Don’t export a parent directory or the root filesystem.
Firewall rules and NFS version: Prefer NFSv4, which uses only port 2049. NFSv3 requires opening additional ports for mountd, rpcbind, and statd.
Kerberos: APC’s NFS deploy feature doesn’t support Kerberized NFS mounts. Houston generates the PersistentVolume with only a server and path — it doesn’t set mountOptions, so there’s no way to request sec=krb5 through this feature. If your NFS server requires Kerberos, this deploy mechanism isn’t currently compatible with it.
Audit logging: Enable NFS server audit logging if required for compliance.

Alternative: Git-sync deploys

If NFS infrastructure isn’t available, consider git-sync deploys which pull DAGs from a Git repository. Git-sync provides:

Version control for Dags.
No external storage infrastructure required.
Webhook-based or polling synchronization.
Branch-based deployment strategies.

Overview

Architecture

Upgrade to Astro Private Cloud

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

Houston API

Platform Observability

Release notes

Version compatibility

Reference

Deploy DAGs with NFS

How NFS deploys work

Implementation considerations

Prerequisites

Enable NFS volume storage

Provision an NFS volume

Configure NFS for a Deployment

Deploy Dags to NFS volume

Direct copy

Using kubectl

CI/CD integration

Sync from cloud storage

Verify NFS configuration

Troubleshooting

Dags aren’t appearing

Permission denied errors

Stale file handle

Network connectivity issues

Security considerations

In your Kubernetes cluster

On your NFS server

Alternative: Git-sync deploys

​How NFS deploys work

​Implementation considerations

​Prerequisites

​Enable NFS volume storage

​Provision an NFS volume

​Configure NFS for a Deployment

​Deploy Dags to NFS volume

​Direct copy

​Using kubectl

​CI/CD integration

​Sync from cloud storage

​Verify NFS configuration

​Troubleshooting

​Dags aren’t appearing

​Permission denied errors

​Stale file handle

​Network connectivity issues

​Security considerations

​In your Kubernetes cluster

​On your NFS server

​Alternative: Git-sync deploys

How NFS deploys work

Implementation considerations

Prerequisites

Enable NFS volume storage

Provision an NFS volume

Configure NFS for a Deployment

Deploy Dags to NFS volume

Direct copy

Using kubectl

CI/CD integration

Sync from cloud storage

Verify NFS configuration

Troubleshooting

Dags aren’t appearing

Permission denied errors

Stale file handle

Network connectivity issues

Security considerations

In your Kubernetes cluster

On your NFS server

Alternative: Git-sync deploys