🚀 Airflow Copilot — Deployment Modes Overview

This guide compares the three deployment options for Airflow Copilot:

🧪 Local Development (Docker + Ngrok)
🧱 Docker-based Production Deployment
☁️ Kubernetes-based Cloud Deployment

Note: The following sections assume you’ve already completed Azure Bot creation and installed the Copilot app in Microsoft Teams.

All options use the same .env file format for managing environment variables.

📊 Comparison Table

Aspect	Local (Dev)	Docker (Prod)	Kubernetes (Prod)
Deployment Method	Docker Compose	Docker Compose	Kubernetes YAML manifests
Public Access	Ngrok Tunnel	Reverse Proxy (e.g., NGINX)	Ingress / LoadBalancer + DNS
Bot Endpoint Setup	Auto-patched via `update_bot.sh`	Manual or scripted	Manual or scripted
PostgreSQL	Host or Container DB (ephemeral)	External DB (e.g., RDS)	Cloud-hosted DB or StatefulSet
TLS/HTTPS	Via Ngrok	Setup via NGINX	TLS via cert-manager
Secrets Management	`.env` file	`.env` file	Kubernetes Secrets
Scaling	Not applicable	Manual	HPA or KEDA
Logging & Monitoring	Local stdout	Volume/log driver	Centralized (ELK / Prometheus / Grafana)

🛠 Prepare Environment Variables

All deployment types rely on a single .env file.

.env

# ───────── LLM (pick ONE provider) ─────────
LLM_MODEL_PROVIDER_NAME=OpenAI
LLM_MODEL_NAME=gpt-4o
OPENAI_API_KEY=your-openai-api-key

# Google example:
# LLM_MODEL_PROVIDER_NAME=Google_Genai
# LLM_MODEL_NAME=gemini-2.5-flash
# GOOGLE_GENAI_API_KEY=ai-...

# Anthropic example
# LLM_MODEL_PROVIDER_NAME=Anthropic
# LLM_MODEL_NAME=claude-3-5-sonnet-20240620
# ANTHROPIC_API_KEY=ai-...

# Groq example
# LLM_MODEL_PROVIDER_NAME=Groq
# LLM_MODEL_NAME=groq-1
# GROQ_API_KEY=ai-...

# ───────── Azure Bot / Service Principal ─────────
MICROSOFT_APP_ID=your-bot-id
MICROSOFT_APP_PASSWORD=bot-secret
AZURE_CLIENT_ID=your-client-id
AZURE_BOT_NAME=your-bot-name # e.g., Airflow-Copilot
AZURE_CLIENT_SECRET=your-spn-secret
AZURE_TENANT_ID=your-tenant-id
RESOURCE_GROUP=your-resource-group


# ───────── Airflow REST API ─────────
AIRFLOW_BASE_URL=your-airflow-url # e.g., http://localhost:8080/
AIRFLOW_AUTH_STRATEGY=per_user # default is 'per_user', can also be 'centralized'

# ───────── Postgres connection ─────────
# Pass Airflow Postgres connection string here if you are using same postgres for Airflow and Copilot
# If you are using a different Postgres instance, update the connection string accordingly.
# Example:
# DB_USER=airflow
# DB_PASSWORD=airflow
# DB_HOST=host.docker.internal
# DB_PORT=5432  
DB_URI=postgresql://<your-db-user>:<your-db-password>@<your-db-host>:<your-db-port>/<your-db-name>

# ───────── Misc ─────────
FERNET_SECRET_KEY=your-fernet-secret-key # Generate using: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
MIN_MSG_TO_RETAIN=10 #default is 10
# Minimum number of messages to retain in the conversation history before summarization
MIN_MSG_TO_SUMMARIZE=10 #default is 10
# ───────── Summarization LLM ─────────
SUMMARIZATION_LLM_MODEL_PROVIDER_NAME=OpenAI # or OpenAI, Anthropic, Google_Genai
SUMMARIZATION_LLM_MODEL_NAME=gpt-4o # or gpt-4o, claude-3-5-sonnet


# ───────── Optional ─────────
# If you want to use ngrok for local development, set your ngrok authtoken
NGROK_AUTHTOKEN=your-ngrok-authtoken # Optional, if you want to use ngrok for local development

1️⃣ Local Deployment (Docker + Ngrok)

Recommended for development and testing.
Ngrok exposes a public endpoint for Microsoft Teams.
The bot-updater service automatically updates the Azure Bot messaging endpoint using Azure Service Principal.

Architecture

Local Deployment

Steps:

Follow the full Getting Started guide.

2️⃣ Docker-Based Production Deployment

Uses the same Copilot Docker image, but with a hardened .env.
A reverse proxy like NGINX is used to expose the service over TLS.
PostgreSQL must be externally hosted (e.g., RDS, Cloud SQL).
Azure Bot messaging endpoint must be updated manually.

Prerequisites (Docker Production)

In addition to the general requirements listed earlier, the following components are essential for deploying Airflow Copilot in a production environment using Docker Compose:

Apache Airflow Instance
A running Airflow instance (version ≥ 2.5.0) with the REST API enabled and accessible to Copilot.
Persistent PostgreSQL Database
A reliable PostgreSQL instance (self-hosted or managed, e.g., AWS RDS, Azure DB) to store conversation history and LangGraph checkpoints.
Reverse Proxy (e.g., NGINX or Traefik)
Required to securely expose FastAPI (/api/messages) over HTTPS.

💡 You may also want to integrate with a certificate manager like Let's Encrypt for automatic HTTPS.

Architecture

Prod Docker

Steps:

Prepare .env file: Populate your .env file with required values. Skip any variables that used only in local mode.
Reference Environment Variables

Create docker-compose.prod.yml

docker-compose.prod.yml

docker-compose.prod.yml

version: "3.9"

# ──────────────── Common env-file anchor ────────────────
x-env: &envfile
  env_file: .env  # Loaded from .env or .env.local

networks:
  airflow:
    driver: bridge

services:

  ###########################################################################
  # 1) Copilot – FastAPI + LangGraph agent (handles DB init internally)
  ###########################################################################
  copilot:
    image: thedatacarpenter/airflow-copilot:latest
    container_name: copilot
    restart: unless-stopped
    ports:
      - "3978:3978"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3978/health"]
      interval: 30s
      timeout: 60s
      retries: 5
    environment:
      USE_DOTENV: false

      # LLM / model
      LLM_MODEL_PROVIDER_NAME: ${LLM_MODEL_PROVIDER_NAME}
      LLM_MODEL_NAME: ${LLM_MODEL_NAME}

      # Provider keys
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      GOOGLE_GENAI_API_KEY: ${GOOGLE_GENAI_API_KEY}
      # GROQ_API_KEY: ${GROQ_API_KEY}
      # ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}

      # Summarization thresholds
      MIN_MSG_TO_SUMMARIZE: ${MIN_MSG_TO_SUMMARIZE:-10}
      MIN_MSG_TO_RETAIN: ${MIN_MSG_TO_RETAIN:-10}

      # Airflow connection
      AIRFLOW_AUTH_STRATEGY: ${AIRFLOW_AUTH_STRATEGY:-per_user}
      AIRFLOW_USER_NAME: ${AIRFLOW_USER_NAME:-airflow}
      AIRFLOW_USER_PASSWORD: ${AIRFLOW_USER_PASSWORD:-airflow}
      AIRFLOW_BASE_URL: ${AIRFLOW_BASE_URL}

      # Summarization model
      SUMMARIZATION_LLM_MODEL_PROVIDER_NAME: ${SUMMARIZATION_LLM_MODEL_PROVIDER_NAME}
      SUMMARIZATION_LLM_MODEL_NAME: ${SUMMARIZATION_LLM_MODEL_NAME}

      # Database & Bot
      DB_URI: ${DB_URI}
      MICROSOFT_APP_ID: ${MICROSOFT_APP_ID}
      MICROSOFT_APP_PASSWORD: ${MICROSOFT_APP_PASSWORD}

      # Encryption
      FERNET_SECRET_KEY: ${FERNET_SECRET_KEY}
    volumes:
      - .env:/app/src/.env:ro
    networks: [airflow]

Start Services
```
# Create Docker network (once)
docker network create airflow

# Start all services
docker compose -f docker-compose.prod.yml up -d
```
4. Expose via Reverse Proxy:

Configure NGINX or similar to route https://copilot.yourdomain.com/api/messages → http://copilot:3978. Then update your Azure Bot endpoint in the portal:
```
    https://copilot.yourdomain.com/api/messages
```
5. You're Done !!!

3️⃣ Kubernetes Deployment (Cloud)

Ideal for enterprise-scale deployments.
Public domain and TLS managed through Ingress + cert-manager.
Environment variables passed securely as Kubernetes Secrets.

See: production-deployment.md

Prerequisites (Kubernetes Production)

Kubernetes Cluster
A working K8s cluster (e.g., AKS, EKS, GKE, or self-hosted).
kubectl access with appropriate permissions to apply resources.
Docker Registry Access
A container registry (e.g., Docker Hub, GHCR, ECR) to host and pull the Copilot Docker image.
Ensure the image is pushed and accessible from within the cluster.
Azure Bot App with Teams Channel
A registered Azure Bot with Microsoft App ID & Password.
The bot’s messaging endpoint must point to your public Copilot Ingress URL.
Cloud-hosted PostgreSQL (or StatefulSet)
A production-grade PostgreSQL instance (e.g., RDS, Cloud SQL) for storing Copilot state and summaries.
Alternatively, deploy it as a StatefulSet inside Kubernetes with persistent volumes.
Ingress Controller (e.g., NGINX)
Required to expose the Copilot FastAPI service publicly.
Enables domain-based access and TLS termination.
TLS Certificates (Recommended)
Use cert-manager with Let’s Encrypt to secure your public domain via HTTPS.
Update the Azure Bot endpoint to use the secured URL.
Secrets Management
Store sensitive environment variables as Kubernetes Secrets.
Use sealed secrets, HashiCorp Vault, or native K8s secrets based on your security policy.

💡 All required configuration values (API keys, credentials, endpoints) should be declared in a shared .env file and converted to Kubernetes secrets during deployment.

Architecture

Kubernetes Deployment

Steps:

Prepare Environment Variables: Create a .env file (as mentioned above) and populate it with the required environment variables. You may omit any variables that are specifically needed for local deployments only. For a complete reference, see Environment Variables for variable details.

Generate Kubernetes Secret

    kubectl create namespace airflow-copilot
    kubectl -n airflow-copilot create secret generic airflow-copilot-env --from-env-file=.env

3. Create Deployment Manifest

airflow-copilot-deployment.yaml

airflow-copilot-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-copilot
  namespace: airflow-copilot
spec:
  replicas: 2
  selector:
    matchLabels:
      app: airflow-copilot
  template:
    metadata:
      labels:
        app: airflow-copilot
    spec:
      containers:
        - name: airflow-copilot
          image: thedatacarpenter/airflow-copilot:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3978
          envFrom:
            - secretRef:
                name: airflow-copilot-env
          readinessProbe:
            httpGet:
              path: /ready
              port: 3978
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 3978
            initialDelaySeconds: 15
            periodSeconds: 10

Create Ingress (Optional)

airflow-copilot-ingress.yaml

airflow-copilot-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: airflow-copilot-ingress
  namespace: airflow-copilot
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
    - hosts: ["airflow-copilot.example.com"]
      secretName: airflow-copilot-tls
  rules:
    - host: airflow-copilot.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: airflow-copilot-svc
                port: { number: 80 }

Apply All Resources

    kubectl apply -n airflow-copilot -f airflow-copilot-deployment.yaml
    kubectl apply -n airflow-copilot -f airflow-copilot-service.yaml
    kubectl apply -n airflow-copilot -f airflow-copilot-ingress.yaml

7. Update Azure Bot Endpoint Update your Azure Bot messaging endpoint to the public URL exposed via Ingress:

```url
https://airflow-copilot.yourdomain.com/api/messages

```
See [**Azure Bot Setup**](../quickstart/azure_bot.md) for more details.

8. You're Done !!!

📦 Deployment Summary

Mode	Ideal For	Public Endpoint	DB Storage
Local	Testing/dev	Ngrok	Host/local
Docker	Small teams	Reverse Proxy/Nginx	Cloud DB
Kubernetes	Enterprise Prod	DNS + TLS Ingress	Managed DB