# Deployment Runbook — `quber table` cloud job (QUE-266)

Stands up the per-document Fargate job that runs the equivalent of
`quber table <doc> --review`, reading a source PDF from S3 (`qubera-docs`) and
writing artifacts to S3 (`qubera-extracts`). The extraction path is CPU-only
(set-of-mark: poppler + Camelot + Anthropic Haiku), so there is no GPU and
nothing runs on RunPod.

All resource names below are exactly what the Terraform and CI expect.

## What gets created

| Resource | Name |
| --- | --- |
| ECR repository | `quber` |
| Output bucket | `qubera-extracts` (input `qubera-docs` already exists; we add its notification) |
| ECS cluster / task def | `quber` / family `quber-table` |
| Trigger Lambda | `quber-trigger` |
| CI role (GitHub OIDC) | `quber-github-actions` |
| Task roles | `quber-task-execution`, `quber-task` |
| Secrets (SSM SecureString) | `/quber/ANTHROPIC_API_KEY`, `/quber/LOGFIRE_TOKEN`, `/quber/ADE_API_KEY` |
| Log groups | `/quber/table-job`, `/aws/lambda/quber-trigger` |

Region is `us-east-1` throughout (qubera-docs lives there, so no cross-region
transfer).

## Prerequisites

- AWS admin credentials in your shell — the apply creates IAM roles and the
  GitHub OIDC provider. Confirm with `aws sts get-caller-identity`.
- Tooling: `terraform >= 1.10` (native S3 state lock), `aws` CLI, `git`.
- `qubera-docs` already exists in `us-east-1` (Terraform reads it as a data
  source; it is not created here).
- A **default VPC** exists in `us-east-1` — the task runs on its public subnets
  with a public IP, reaching ECR/S3/Anthropic through the internet gateway (no
  NAT gateway, so no ~$32/month cost).

Run the [pre-flight checks](#pre-flight-checks) below before step 1 to confirm
credentials, the default VPC, and its subnets.

## 1. Bootstrap the Terraform state bucket (one-time)

```bash
aws s3api create-bucket --bucket qubera-tfstate --region us-east-1
aws s3api put-bucket-versioning --bucket qubera-tfstate \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket qubera-tfstate \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}'
```

## 2. Provision infrastructure

```bash
cd deploy/terraform
cp backend.hcl.example backend.hcl        # bucket=qubera-tfstate, region=us-east-1
terraform init -backend-config=backend.hcl
terraform plan                            # review the plan
terraform apply
terraform output                          # note github_actions_role_arn, ecr_repository_url, ...
```

If the account **already has** a GitHub OIDC provider, import it first (a
duplicate URL errors):

```bash
terraform import aws_iam_openid_connect_provider.github \
  arn:aws:iam::<ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com
```

## 3. Set the secret values (out of band — never in Terraform state)

Terraform creates the parameters with a placeholder and ignores value drift, so
set the real values directly:

```bash
aws ssm put-parameter --name /quber/ANTHROPIC_API_KEY --type SecureString --value '<key>'   --overwrite
aws ssm put-parameter --name /quber/LOGFIRE_TOKEN      --type SecureString --value '<token>' --overwrite
aws ssm put-parameter --name /quber/ADE_API_KEY        --type SecureString --value '<key>'   --overwrite
```

## 4. Wire GitHub Actions

- Add the repository secret **`AWS_ACCOUNT_ID`** = your 12-digit account id.
  `deploy.yml` builds the role ARN from it and assumes `quber-github-actions`
  (created in step 2) via OIDC. No AWS access keys are stored.

## 5. First image build + deploy

```bash
git tag v0.1.0
git push origin v0.1.0
```

`deploy.yml` then assumes the OIDC role, builds `deploy/Dockerfile`, pushes
`quber:<git-sha>` to ECR, and registers a new `quber-table` task-definition
revision pointing at that image.

No-CI alternative (build and push locally):

```bash
ECR=$(cd deploy/terraform && terraform output -raw ecr_repository_url)
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin "${ECR%/*}"
docker build -f deploy/Dockerfile -t "$ECR:v0.1.0" .
docker push "$ECR:v0.1.0"
# then register a task-def revision pointing at $ECR:v0.1.0
```

## 6. Smoke test

Upload a PDF to trigger the Lambda, then watch the logs and check the output:

```bash
aws s3 cp sample.pdf s3://qubera-docs/inbound/test/sample.pdf

aws logs tail /aws/lambda/quber-trigger --follow
aws logs tail /quber/table-job --follow

aws s3 ls s3://qubera-extracts/test/sample/
# expect: sample.tables.json, sample.review.html, sample.boxed.pdf
```

## Ordering gotchas

- **Set the secrets (step 3) before any task runs.** Without
  `ANTHROPIC_API_KEY` the container cannot call Haiku.
- **Run the first deploy (step 5) before uploading real PDFs.** The task
  definition Terraform creates points at `quber:latest`, which does not exist
  until the first push; the deploy registers a revision with a real image. A
  trigger before that cannot start a task.
- **The `qubera-docs` notification is Terraform-managed.** If that bucket
  already has event notifications, the apply replaces them.

## Troubleshooting

### Pre-flight checks

Run these before step 1 to catch the common blockers:

```bash
# 1. Credentials resolve and point at the intended account
aws sts get-caller-identity

# 2. A default VPC exists in us-east-1 (the task runs on its public subnets)
aws ec2 describe-vpcs --filters Name=isDefault,Values=true --region us-east-1 \
  --query 'Vpcs[].VpcId' --output text

# 3. That VPC has subnets (Terraform uses all subnets in it).
#    Substitute the VPC id printed by step 2.
aws ec2 describe-subnets --region us-east-1 \
  --filters Name=vpc-id,Values=<DEFAULT_VPC_ID> \
  --query 'Subnets[].SubnetId' --output text
```

- Step 2 empty (no default VPC): run `aws ec2 create-default-vpc --region us-east-1`,
  or make the VPC/subnets Terraform variables and point them at an existing VPC
  (private subnets then need a NAT gateway or ECR + S3 VPC endpoints).
- Step 3 empty (subnets deleted): recreate the default subnets, or supply your
  own subnet ids as above.

### Common failures

- **`apply` fails creating the OIDC provider (`EntityAlreadyExists`).** The
  account already has the GitHub provider. Import it, then re-apply:
  `terraform import aws_iam_openid_connect_provider.github arn:aws:iam::<ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com`
- **Task stops immediately with an image-pull error.** No image is published
  yet. Run step 5 (tag a release) before relying on the trigger — the initial
  task def points at `quber:latest`, which does not exist until the first push.
- **Task fails calling the model / missing `ANTHROPIC_API_KEY`.** The SSM
  parameters still hold the `REPLACE_ME` placeholder. Set them (step 3).
- **No task starts on upload.** Check the Lambda log
  (`aws logs tail /aws/lambda/quber-trigger`): a non-`.pdf` key or an
  already-populated output prefix is skipped by design; a `RunTask` failure is
  logged with the AWS error.

## Playground

The hosted playground is the second workload in this stack: the FastAPI app
and its UI as a Fargate service behind an HTTPS load balancer at
`playground.qubera.ai`, an RDS PostgreSQL instance with pgvector as its
store, and the extracts bucket as the durable home of every document's
files. It scales to zero when idle. Everything is Terraform in this
directory (`rds.tf`, `playground.tf`) and applies with the same
`terraform plan` / `terraform apply` as the table job.

### What gets created

| Resource | Name |
| --- | --- |
| Database | RDS `quber-playground` (PostgreSQL 17, `db.t4g.micro`, pgvector), security group `quber-playground-db` |
| Image repository | ECR `quber-playground` (keeps the last 3 images) |
| Service | ECS service `quber-playground` in cluster `quber`, family `quber-playground`, security group `quber-playground-task` |
| Load balancer | ALB `quber-playground`, target groups `quber-playground-app` and `quber-playground-wake`, security group `quber-playground-alb` |
| Hostname | `playground.qubera.ai` in the existing `qubera.ai` zone, ACM certificate validated through it |
| Wake function | Lambda `quber-playground-wake`, the sign-in and startup page while no task is healthy |
| Idle scale-in | Alarm `quber-playground-idle` on the app's signed-in request count, plus a five-minute check |
| Roles | `quber-playground-execution`, `quber-playground-task`, `quber-playground-wake`; the CI role gains the `ci-playground` policy |
| Secrets (SSM SecureString) | `/quber/POSTGRES_PASSWORD`, `/quber/WORKOS_API_KEY`, `/quber/WORKOS_CLIENT_ID`, `/quber/WORKOS_ORGANIZATION_ID`, `/quber/PLAYGROUND_SESSION_SECRET`, `/quber/TYPESAFE_API_KEY`; `/quber/PLAYGROUND_LOGIN_USER` and `/quber/PLAYGROUND_LOGIN_PASSWORD` are kept only for rolling back to the shared login | |
| Log groups | `/quber/playground`, `/aws/lambda/quber-playground-wake` |

`terraform.tfvars` needs `operator_cidr`: the developer host's address as a
`/32`, admitted to the database for the schema restore and for running the
local playground against RDS. Change it when the home IP changes.

### Database

The instance is created with a throwaway master password and the real one is
rotated onto it out of band, so the real password lives only in Parameter
Store. After the first apply:

```bash
aws ssm put-parameter --name /quber/POSTGRES_PASSWORD --type SecureString \
  --value "$(openssl rand -base64 30 | tr -d '/+=' | cut -c1-32)" --overwrite
aws rds modify-db-instance --db-instance-identifier quber-playground --apply-immediately \
  --master-user-password "$(aws ssm get-parameter --name /quber/POSTGRES_PASSWORD \
    --with-decryption --query Parameter.Value --output text)"
```

Enable the extension and load the schema. The dump is taken inside the local
pgvector container because the host's `pg_dump` is older than the container's
server; the restore runs from the same container:

```bash
RDS=$(cd deploy/terraform && terraform output -raw playground_db_endpoint)
export PGPASSWORD=$(aws ssm get-parameter --name /quber/POSTGRES_PASSWORD --with-decryption --query Parameter.Value --output text)
psql -h "$RDS" -U quber -d quber_rag -c 'create extension if not exists vector'
docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" pgvector-db \
  pg_dump -U quber -d quber_rag -n ade_playground -Fc -f /tmp/ade_playground.dump
docker exec -e PGPASSWORD pgvector-db \
  pg_restore -h "$RDS" -U quber -d quber_rag --no-owner --no-privileges -Fc /tmp/ade_playground.dump
```

The local playground runs against RDS with `POSTGRES_HOST=$RDS` and
`POSTGRES_PASSWORD` set to the Parameter Store value; nothing else changes.

### Login

People sign in through WorkOS, in the Playground Staging environment, and
only members of the organization in `/quber/WORKOS_ORGANIZATION_ID` get in.
The app turns the sign-in on when all four parameters are set, and refuses
to start on a partial set. Take the API key and client ID from the WorkOS
dashboard (the environment's API Keys page), and the organization id from
its Organizations page:

```bash
aws ssm put-parameter --name /quber/WORKOS_API_KEY            --type SecureString --value '<sk_...>'     --overwrite
aws ssm put-parameter --name /quber/WORKOS_CLIENT_ID          --type SecureString --value '<client_...>' --overwrite
aws ssm put-parameter --name /quber/WORKOS_ORGANIZATION_ID    --type SecureString --value '<org_...>'    --overwrite
aws ssm put-parameter --name /quber/PLAYGROUND_SESSION_SECRET --type SecureString --value "$(openssl rand -hex 32)" --overwrite
```

A person gets access by being invited to that organization from the WorkOS
dashboard; sign-up is off, so nobody can create an account on their own.
Removing them from the organization takes effect when their cookie next
expires, within two hours.

The WorkOS dashboard also has to list the callbacks and sign-out addresses
for every site people sign in from: `https://playground.qubera.ai/callback`
as the redirect URI and `https://playground.qubera.ai/` as the sign-out URI.
WorkOS refuses a sign-in that returns anywhere else.

`/quber/PLAYGROUND_LOGIN_USER` and `/quber/PLAYGROUND_LOGIN_PASSWORD` belong to
the shared login this replaced. Nothing reads them now; they stay so that
redeploying an image from before the change still signs in.

To run the sign-in on the developer host, keep the four values in a file
other than `.env` (a partial set there stops every local run), load it, and
serve on an address registered in the WorkOS dashboard:

```bash
set -a; . ./.env.workos; set +a
uv run quber playground --host 192.168.1.237 --port 8101
```

### Chunk ranker key

The task also carries the TypeSafe key for the playground's Jev chunk ranker.
The ranker is off until `QUBER_PLAYGROUND_RANKER` is set to `jev` on the task,
but the key is set now so that switch needs nothing else:

```bash
aws ssm put-parameter --name /quber/TYPESAFE_API_KEY --type SecureString --value '<key>' --overwrite
```

### Image and deploy

```bash
git tag playground-v0.1.0
git push origin playground-v0.1.0
```

`deploy-playground.yml` builds `deploy/playground.Dockerfile`, pushes it to
`quber-playground`, registers a task-definition revision and points the
service at it. The service keeps its desired count, so a scaled-to-zero
playground picks the new image up on its next wake. The no-CI alternative:

```bash
ECR=$(cd deploy/terraform && terraform output -raw playground_ecr_repository_url)
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin "${ECR%/*}"
docker build -f deploy/playground.Dockerfile -t "$ECR:latest" .
docker push "$ECR:latest"
```

The task definition Terraform creates points at `:latest`, so the first push
must carry that tag.

### Documents in the bucket

Every document's files live under `s3://qubera-extracts/documents/<doc_key>/`
with the same names they have in the local data directory. The app copies
each upload's files there as its stages finish and fetches a served PDF back
when a task no longer has it. To publish the local library once:

```bash
QUBER_PLAYGROUND_ARTIFACTS_URI=s3://qubera-extracts/documents/ \
  uv run python -m quber.playground.storage
```

### Scale to zero, and waking

The service's desired count is 0 when idle. The HTTPS listener has one rule
for every path and the wake function owns where it points: at the app's
target group while a task is healthy, at the function itself otherwise. The
load balancer cannot make that choice on its own -- a forward to an empty
target group is a plain 503 -- so the function is invoked by everything that
changes the answer.

Only a person wakes the service. While no task runs, the function is the
sign-in: it sends the browser to WorkOS, finishes the sign-in WorkOS sends
back, checks the organization against the same Parameter Store values as the
app, and sets the same session cookie (the sign-in, the check and the cookie
live in `src/quber/playground/session.py`, packaged into the function beside
`wake.py`). A completed sign-in, or a request that
already carries a valid session, starts a task and returns a startup page
that polls `/healthz`; the first poll after the task turns healthy throws
the switch to the app and the page reloads. Anything else gets the sign-in page
or a redirect to it and starts nothing, which is what keeps internet scanners
from running the service around the clock: they reach the hostname every
few minutes, and before this gate each of them woke a task.

Idle is measured on the count of requests that carried a valid session,
published by the login gate once a minute as
`Quber/Playground AuthenticatedRequests`; a sign-in at the function counts
as one. The alarm's change to ALARM after `playground_idle_minutes` (30)
without one, and any task of the service stopping, both reach the function
through EventBridge and throw the switch back. The five-minute check scales
in a task that woke into an alarm already in ALARM, unless there was a
signed-in request in the last ten minutes or the task is under five minutes
old and its person has not reached it yet. To wake or stop the service by
hand:

```bash
aws ecs update-service --cluster quber --service quber-playground --desired-count 1
aws ecs update-service --cluster quber --service quber-playground --desired-count 0
```

The health route reports readiness: 200 only once the embedding model is
loaded and the database answers. The service's health check grace period
covers the boot, so ECS does not replace a task while it loads the model.

## Teardown

```bash
aws s3 rm s3://qubera-extracts --recursive   # empty the bucket first
cd deploy/terraform && terraform destroy
# The database has deletion protection on and takes a final snapshot; turn
# protection off in rds.tf first if the destroy is meant to include it.
```