--- name: rj-reit-sweep description: Quarterly RJ REIT workflow against the playground RAG. Use when loading a new quarter's REIT filings from s3://qubera-docs/rj.reit/ into the document library, preserving scan-output JSONs to S3, or filling the metric sweep workbook (per-ticker tabs, Prompt/10Q/Deck columns) with answered values. allowed-tools: Bash, Read --- # RJ REIT Quarterly Sweep The repeatable version of the workflow first run for Q2 2026: load the quarter's filings into the playground RAG, preserve every document's scan output to S3, and fill the metric sweep workbook by answering each in-scope prompt against the company's 10-Q and deck. ## Prerequisites - The playground server is running: `uv run quber playground` (port 8101). Every script takes `--base` if it runs elsewhere. - AWS credentials with access to `s3://qubera-docs/` (the scripts shell out to `aws s3 cp`). - Run every script with `uv run python` from the repository root so `quber.settings` and the project environment resolve. - Each script has `--dry-run` and `--help`. Dry-run first when in doubt. ## S3 layout contract - Source PDFs live under `s3://qubera-docs/rj.reit//`, one uppercase ticker folder per issuer, filenames unchanged from delivery. - Preserved scan output lives under `s3://qubera-docs/rj.reit/scan-output//`, holding the hash-keyed artifact JSONs. - Filled sweep workbooks are saved as new objects next to their source (`-filled.xlsx`), never overwriting the source workbook. ## Stage 1 — Load the quarter's filings Build a manifest CSV with the columns `s3_uri,folder,filing_type,year,period,title`, one row per document. Folder is the uppercase ticker; filing_type is `10-Q`, `99-1`, `99-2`, `10-K`, or free-form; title follows the house style, e.g. "Q2 2026 Form 10-Q". A document whose filename does not state its ticker or type must be attributed by reading its cover page, never guessed. ```bash uv run python .claude/skills/rj-reit-sweep/scripts/load_documents.py manifest.csv --dry-run uv run python .claude/skills/rj-reit-sweep/scripts/load_documents.py manifest.csv ``` Every row is submitted through the upload API with its metadata and runs the fusion pipeline (two at a time; roughly 2-3 seconds per page). The server hashes the bytes and refuses exact duplicates, so re-running a manifest over already-loaded documents skips them — that is the supported way to top a quarter up. The script watches every job to done or failed and exits nonzero on any failure. Verify: `GET /api/documents` shows the new rows with complete filing labels and no unexpected possible-duplicate flags. ## Stage 2 — Preserve scan output ```bash uv run python .claude/skills/rj-reit-sweep/scripts/preserve_scan_output.py --dry-run uv run python .claude/skills/rj-reit-sweep/scripts/preserve_scan_output.py ``` Uploads every library document's artifact JSONs (unified parse, tables, fusion report, scanned tables, figures, charts, ADE parse where present) to the scan-output prefix. Idempotent; re-runs refresh the copies. A document with no artifacts on disk is reported, never silently dropped. ## Stage 3 — Fill the sweep workbook The workbook has one tab per ticker plus a Summary tab. Each metric row carries Metric, Found, Quarter, a Prompt formula, and blank 10Q and Deck columns. The rules the script enforces: - In scope: Found is `xbrl` or `docling (non-xbrl tool)`. `not_reported` rows are left untouched. - Prompt: computed as the sheet's formula defines it — `Find {Metric}. Return {Quarter} value`. - Recorded value: a scalar's printed value; `#not found#` for an unanswerable; `ERROR: ` for a failed question. An in-scope row with an empty Metric cell is reported and skipped. - Provenance: each answered row's cited sources land in `10Q Provenance` and `Deck Provenance` columns, appended after the sheet's existing headers so no formula column shifts (created if absent). A table cell reads page, table, row, column (1-based row and column; the table keeps the parse's own number); a figure value reads page, chart, segment; a whole table standing alone reads page and table number. Distinct sources only: a cited whole table or line item is dropped when a cited cell in the same table is present, and repeated renderings collapse to one. The Comment column is never written — it belongs to the sheet's readers. - A tab whose ticker lacks a loaded 10-Q/99-2 pair for the target quarter is reported as skipped. - A batch the server loses mid-run (a restart wipes its in-memory registry) raises a named error; the per-ticker checkpoint holds every ticker finished before the loss, so recovery resumes from there rather than rerunning the sweep. ```bash uv run python .claude/skills/rj-reit-sweep/scripts/run_sweep.py \ s3://qubera-docs/rj.reit/.xlsx --period Q2 --year 2026 --dry-run uv run python .claude/skills/rj-reit-sweep/scripts/run_sweep.py \ s3://qubera-docs/rj.reit/.xlsx --period Q2 --year 2026 ``` The filled copy checkpoints after each ticker and lands beside the source as `-filled.xlsx`. Expect roughly 90 seconds per ticker-document batch; the Q2 2026 first pass filled 824 cells across 8 tickers in 26 minutes. `--tickers BRSP,KREF` scopes the run to those tabs only; other tabs are left untouched in the filled copy. A name matching no tab aborts before anything is asked or written, so a typo cannot produce a silently partial run. The playground server watches the whole checkout when run with `--reload`, and its batch jobs live in server memory. Any file change during a sweep — an edit, a `git pull` — restarts the server and kills the in-flight batch; the sweep then stops with a named "batch lost by the server" error. Hold all edits and pulls until the run finishes; recovery is a plain rerun, the cache and checkpoint make the finished portion free. ### Answer cache Answers persist in `ade_playground.sweep_answers`, keyed by the document's content-derived `doc_key`, the exact prompt text, and the answering model. The sweep always consults the cache: a question with an open row reuses its stored value and provenance, and only misses reach the batch API, so a rerun of an unchanged workbook costs no inference. Cached `#not found#` answers are reused like values; error rows never are. Rows are append-only and versioned — a fresh answer closes the old row (`valid_to`) and inserts a new open one, so history is never lost. There is no bypass flag. To re-measure anything — one stubborn miss, a document, everything — close the rows and rerun: ```sql UPDATE ade_playground.sweep_answers SET valid_to = now() WHERE AND valid_to IS NULL; ``` Invalidation is otherwise automatic: an edited Metric changes the prompt text, a changed document changes its `doc_key`, and a model bump changes the key. A change none of those catch (the answer agent's system prompt or retrieval parameters under the same model name) needs a manual close. A completed run's filled workbook can seed the cache, one open row per answered cell, prompts rebuilt exactly as the sweep builds them. Existing open rows are never touched, so re-seeding cannot clobber newer answers: ```bash uv run python .claude/skills/rj-reit-sweep/scripts/seed_answer_cache.py \ s3://qubera-docs/rj.reit/xlsx/-filled.xlsx --period Q2 --year 2026 ``` ## After the run Report counts, never adjectives: documents loaded and failed, artifacts preserved, cells written split into values / `#not found#` / errors, the cached / asked split, and every skipped tab or row with its reason. Comment the results on the quarter's Jira task. Anomalies in the workbook itself (header drift between tabs, in-scope rows with empty metrics) belong in that comment for the sheet owner — fix them in the filled copy only, never in the source.