# RunPod Runbook — docling parse worker Runs the GPU half of the document pipeline: the canonical docling parse (`quber parse`, tuned-financial preset) as a RunPod serverless endpoint. The AWS trigger Lambda submits one job per inbound PDF with presigned S3 URLs; the worker downloads the PDF, parses on CUDA, uploads the three parse artifacts, and finishes with a completion marker whose arrival in S3 drives the join-and-fuse step. The worker holds no AWS credentials — the presigned URLs carry all authorization. The same worker carries a second task for the hosted playground: a job whose input says `task: embed` downloads a chunks file, embeds the texts with the playground's embedding model (bge-large-en-v1.5, baked into the image), uploads the vectors and a completion marker, all through presigned URLs like the parse. The playground's `quber.playground.gpu` module is the only caller. RunPod's model has three layers. The **endpoint** is the stable HTTPS API the Lambda calls; it never changes. A **template** binds one exact image tag and is treated as immutable — one template per released image. **Workers** are containers spawned from whatever template the endpoint currently references, billed per second, scaled to zero when idle. ## What exists | Piece | Value | | --- | --- | | Endpoint | `quber-parse` (`e9d1h9f9oehnr7`), base URL `https://api.runpod.ai/v2/e9d1h9f9oehnr7` | | GPU tier | RTX 5090 first, then the 48GB datacenter Ampere cards (A40, RTX A6000); hosts restricted to CUDA 13.0 drivers to match the image's torch build. The consumer 24GB pools were dropped after they repeatedly placed workers on hosts with drivers too old for the image. | | Workers | 0 active / 3 flex max, 5s idle timeout, FlashBoot off (warm workers on a stale image survived verified-zero drains), 600s execution timeout | | Templates | `quber-parse-worker-`, one per released image | | Image | `ghcr.io/xmandeng/quber-runpod:` (private GHCR package, ~10GB) | | Registry credential | stored in RunPod as `ghcr-quber` (GHCR PAT with `read:packages`) | | API key | `RUNPOD_API_KEY` in the repo `.env`; mirrored to SSM `/quber/RUNPOD_API_KEY` for the Lambdas | | CI | `.github/workflows/runpod-image.yml`, fires on `runpod-v*` tags | The image is self-contained: dependencies, the model weights the preset loads (~550MB), a C compiler for torch's runtime kernel compilation, and an offline-mode flag so a worker never depends on HuggingFace being reachable. A cold start on a fresh host pays a one-time image pull (minutes); warm requests run in seconds. ## Releasing a new worker image The tag is the release decision; everything after it is one command. 1. Merge to `main`. From a checkout on fresh `main`: ```bash git tag runpod-v0.2.1 # next version; never reuse or move a tag git push origin runpod-v0.2.1 ``` CI builds and pushes `ghcr.io/xmandeng/quber-runpod:v0.2.1` (~15 min). Confirm in the repo's Actions tab, or poll GHCR for the tag. 2. Repoint the endpoint — the only manual step: ```bash uv run python runpod/create_endpoint.py \ --image ghcr.io/xmandeng/quber-runpod:v0.2.1 ``` Idempotent: reuses the stored registry credential, creates the per-tag template, moves the endpoint to it, and **refreshes the workers** (drains `workersMax` to 0, restores it). The refresh is not optional politeness — existing warm workers keep running the old image after a template move and will win the race for new jobs, failing them in milliseconds if the job contract changed. 3. Verify: ```bash curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \ https://api.runpod.ai/v2/e9d1h9f9oehnr7/health ``` Workers should show `initializing` (pulling the new image), then `idle`/`ready`. For a full check, drop a small PDF into `s3://qubera-docs/inbound/` and watch its output prefix. First-time provisioning (new account or new endpoint) is the same script with two extra env vars so it can store the pull credential: `GHCR_USERNAME` and `GHCR_PULL_TOKEN` (a long-lived PAT with `read:packages`). Everything else — tier, workers, timeouts — is encoded in the script. ## Job contract The Lambda posts to `/run` with: ```json {"input": { "base": "", "pdf_url": "", "artifact_urls": {"document": "", "confidence": "", "cells": ""}, "marker_url": "", "log_url": "" }} ``` The worker writes the marker strictly after every artifact, so the marker's existence in S3 guarantees the artifacts are whole. On failure the marker carries `status: failed` with the failing stage — failures ride the same channel as successes, and the join Lambda holds the document instead of fusing. A job that dies without writing anything is caught by the one-shot deadline check the Lambda scheduled at submission (execution timeout plus margin); it queries `/status` and raises the same hold-and-alert. ## Held documents A held document means one extraction half failed; nothing was fused. ```bash aws s3 ls s3://qubera-extracts/hold/ --recursive # the queue of held documents aws s3 cp s3://qubera-extracts//hold.json - # why this one was held ``` Release after fixing the cause: delete `hold.json`, the `hold/` pointer, and the failed half's marker from the document's prefix, then re-drive by re-copying the source PDF onto itself: ```bash aws s3 cp s3://qubera-docs/inbound/.pdf s3://qubera-docs/inbound/.pdf \ --metadata-directive REPLACE ``` The trigger Lambda's per-engine dedup re-runs only the half whose results are missing; the completed half is never repeated. ## Routine operations - **Health / queue depth**: the `/health` curl above. - **Worker logs**: the worker mirrors its log to `s3://qubera-extracts//.parse.log` every 5 seconds while a job runs and once more at job end, success or failure. Tail it near-live with `watch -n 5 aws s3 cp s3://qubera-extracts//.parse.log -`. The RunPod console (Serverless → `quber-parse` → Workers, and the Requests tab) still shows container startup output that predates the handler; completed-job records there expire after RunPod's retention window (tens of minutes). - **Spend**: prepaid balance is the spend cap — there is deliberately no other limit. Check with the GraphQL API: `query { myself { clientBalance currentSpendPerHr } }` against `https://api.runpod.io/graphql` (use curl; Python urllib is blocked by Cloudflare on this host, though the `api.runpod.ai` job API accepts it). - **Idle decay**: an endpoint idle for 3 days has `workersMax` cut to 2, and after 7 idle days to 0 — at which point it serves nothing until the cap is raised (`PATCH /v1/endpoints/` with `{"workersMax": 3}`, or rerun the release script). The deadline check makes this failure loud: documents dropped against a decayed endpoint are held, not lost. - **Manual worker refresh** (rarely needed outside releases): ```bash curl -s -X PATCH -H "Authorization: Bearer $RUNPOD_API_KEY" \ -H "Content-Type: application/json" \ https://rest.runpod.io/v1/endpoints/e9d1h9f9oehnr7 -d '{"workersMax": 0}' sleep 20 curl -s -X PATCH -H "Authorization: Bearer $RUNPOD_API_KEY" \ -H "Content-Type: application/json" \ https://rest.runpod.io/v1/endpoints/e9d1h9f9oehnr7 -d '{"workersMax": 3}' ``` ## MIG substitution (GPU drift) RunPod partitions RTX PRO 6000 Blackwell cards into MIG slices and injects them into the serverless capacity categories by default (1g.24gb into 24GB, 2g.48gb into 48GB) — and the scheduler matches by category, so pinning `gpuTypeIds` does not keep them out. The parse pipeline fails on those slices while passing on full Ampere GPUs. The REST API has no field for MIG; the opt-out lives in the GraphQL `gpuIds` string as a negative entry (what the console's Advanced checkbox writes): ``` ADA_32_PRO,AMPERE_48,-NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb,-NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 2g.48gb ``` `create_endpoint.py` re-asserts the exclusion on every run (`ensure_mig_exclusions`). If jobs ever fail again with a GPU name containing "MIG" in the worker log's job-done URL, check `query { myself { endpoints { gpuIds } } }` first. New MIG profiles RunPod introduces need adding to `MIG_EXCLUSIONS` in the script. ## Failure modes seen in production | Symptom | Cause | Fix | | --- | --- | --- | | Jobs COMPLETED in ~130ms, no artifacts, worker log shows a contract KeyError | Stale workers running the previous image after a template move | Refresh workers (now automatic in the release script) | | Worker PUTs rejected with 400 | Presigned URLs signed with SigV2 (boto3 default in some configurations) | The caller must sign with `signature_version="s3v4"` — the Lambda does | | Parse fails `Pipeline StandardPdfPipeline failed`, log shows `InductorError: Failed to find C compiler` | torch compiles kernels at runtime on some documents; image lacked gcc | gcc/libc6-dev are baked into the image | | Parse hangs on HuggingFace at startup | docling pings the HF API even with cached weights | `HF_HUB_OFFLINE=1` is baked into the image | | Join Lambda crashes reading a missing marker (403, not 404) | S3 returns AccessDenied for GETs of missing keys unless the caller has `s3:ListBucket` | The join role carries ListBucket; remember this for any Lambda probing optional keys | | Large documents fail `Pipeline StandardPdfPipeline failed` on the endpoint but parse fine locally; worker log job-done URL shows a MIG GPU | RunPod schedules Blackwell MIG slices into the 24GB category despite GPU pins | MIG exclusion in `gpuIds` (see MIG substitution section); enforced by `create_endpoint.py` |