Deploy Your Own

Full operator guide for deploying an isolated RDE Workflow Engine environment in your AWS account using CDK. Covers infrastructure, authentication, and day-two operations.

What Gets Deployed

The @amzn/ar-metaflow-cdk-constructs package deploys a complete Metaflow platform as a stack-per-concern architecture. Each stack is independently deployable and updateable.

Deploy order: VPC → DNS → Data → Auth → EKS → Ingress

Before You Start

AWS Account Setup

• AWS account provisioned and CDK bootstrapped (cdk bootstrap aws://<ACCOUNT_ID>/<REGION>)
• IAM role IibsAdminAccess-DO-NOT-DELETE available in the target account
• Root DNS zone deployed via RootDnsStack (one-time, in the DNS account)
• Sufficient service quotas for EKS, GPU instances (g5/g6/p4d), and NLB

CLI Tools

CDK Configuration

Ensure your cdk.json includes:

{
  "requireApproval": "never"
}

Define Your Stage

Add an entry to the STAGES map in lib/constants.ts:

export const STAGES: Record<string, StageConfig> = {
  "my-stage": {
    accountId: "123456789012",
    region: "us-west-2",
    isProd: false,
    oidcConfig: {
      clientId: "iss-metaflow-my-stage",
      idpProviderName: "iss-metaflow-my-stage",
      issuerUrl: "https://idp.federate.amazon.com",
    },
  },
};

First-Time Deployment

Authenticate

ada credentials update \
  --account <ACCOUNT_ID> \
  --provider conduit \
  --role IibsAdminAccess-DO-NOT-DELETE \
  --once \
  --profile iss-metaflow-<stage>

Build

brazil-build release

Deploy Stacks

Deploy in order. Each command is idempotent — safe to re-run on failure.

bb cdk deploy <stage>-vpc       # 3-5 min
bb cdk deploy <stage>-dns       # 2-3 min
bb cdk deploy <stage>-data      # 5-8 min
bb cdk deploy <stage>-auth      # 2-3 min  ← record CfnOutputs after this
bb cdk deploy <stage>-eks       # 15-20 min
bb cdk deploy <stage>-ingress   # 5-8 min

# Total: ~30-45 min

Register with Amazon Federate

One-time manual step after the auth stack deploys. Connects Cognito to Amazon's corporate identity provider.

Store the Secret

aws secretsmanager put-secret-value \
  --secret-id <stage-client-id>/federate-client-secret \
  --secret-string '{"FEDERATE_CLIENT_SECRET": "<YOUR_SECRET>"}' \
  --region <region> --profile iss-metaflow-<stage>

Update Cognito

aws cognito-idp update-identity-provider \
  --user-pool-id <USER_POOL_ID> \
  --provider-name <IDP_PROVIDER_NAME> \
  --provider-details '{
    "client_id": "<CLIENT_ID>",
    "client_secret": "<YOUR_SECRET>",
    "authorize_scopes": "openid email profile",
    "attributes_request_method": "GET",
    "oidc_issuer": "https://idp.federate.amazon.com"
  }' \
  --region <region> --profile iss-metaflow-<stage>

Login and Verify

ar-metaflow login --account <ACCOUNT_ID> --region <region>

Source the environment and add to your shell profile:

source ~/.metaflowconfig/env.sh

# Add to ~/.zshrc or ~/.bashrc (one-time)
[ -f ~/.metaflowconfig/env.sh ] && source ~/.metaflowconfig/env.sh

Run a test flow:

python hello_flow.py run                       # local — uses remote metadata
python hello_flow.py --with kubernetes run     # remote — steps run as K8s pods

Metaflow UI

https://<stage>.metaflow.simulation.amazon.dev

Removing a Deployment

Destroy stacks in reverse order. For isProd: true stages, disable termination protection first.

bb cdk destroy <stage>-ingress
bb cdk destroy <stage>-eks
bb cdk destroy <stage>-auth
bb cdk destroy <stage>-data
bb cdk destroy <stage>-dns
bb cdk destroy <stage>-vpc

Common Issues

invalid_client error during login

Federate client secret is missing or incorrect. Re-run FederateSecretStoreCommand + FederateCognitoUpdateCommand from auth stack outputs. Ensure you're using Prod Federate (not Integ).

UI shows “Looking for connection…”

The Metaflow UI backend isn't reachable. Verify the ingress stack deployed, check DNS resolution (nslookup <stage>.metaflow.simulation.amazon.dev), and inspect EKS pods (kubectl get pods -n metaflow).

DNS not resolving after deploy

NS delegation takes 2–5 minutes. Flush local cache: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

cdk deploy hangs silently

Check cdk.json has "requireApproval": "never". Verify ada credentials haven't expired.

Credential error during EKS deploy

EKS takes 15–20 minutes. If ada tokens expire, the stack rolls back. Re-authenticate and re-run — CDK resumes from the last successful state.

401 Unauthorized running flows

JWT tokens expire after ~1 hour. Re-run:

ar-metaflow login --account <ACCOUNT_ID> --region <region>
source ~/.metaflowconfig/env.sh

“Export not found” during deploy

Stacks deployed out of order. Deploy in sequence: VPC → DNS → Data → Auth → EKS → Ingress.

cdk8s instanceof mismatch

Duplicate cdk8s packages in node_modules. Run npm run clean && npm install && npm run build.

Synth fails with “provider name too long”

Cognito identity provider names have a 32-character limit. Shorten your idpProviderName in the stage config.