# Set-of-Mark extraction — working notes

Procedures for working in this subsystem. The focus is how to gather evidence
and where it lives.

## Orient before reasoning

Start by reading `docs/SET_OF_MARK_DESIGN.pdf`. It is the source of truth for
the pipeline. Hold these three authorities in mind as you read code:

- Vision over the gridded page image owns table **identity and region**
  (Stage 2, `grid_locator.py`).
- Camelot over the PDF text layer owns the **values** inside a region.
- Vision over a cropped table image owns **structure** (columns, headers).

A box is the vision model's call, then refined by a deterministic geometry chain:
`tighten_region`, then `pad_boxes`, then two declash passes, `declash_stacked`
followed by `declash_side_by_side`. The chain refines a region; it does not
decide where a table starts or ends.

## To diagnose a wrong table boundary

Work from the data, so you can tell whether the model drew the box or the
geometry chain moved it.

1. Read `som_region` from the run's `*.tables.json`. It is the final box,
   normalized 0..1 with the page top-left as origin, `(x1, y1, x2, y2)`.
2. Pull the model's raw flags from that run's LangSmith `grid_locate` trace (see
   "To find a run's traces"). The flags are the literal grid spans the model
   reported: `row_start`, `row_end`, `col_start`, `col_end`.
3. Convert the flags to a normalized box and compare to `som_region`. The grid
   is 36 rows by 12 columns, rows 1-based and columns A=0 .. L=11, both
   inclusive. Top is `(row_start - 1) / 36` and bottom is `row_end / 36`. Left
   is `col_start_index / 12` and right is `(col_end_index + 1) / 12`
   (`grid_region_norm` in `agents/grid_locator.py`).
4. Read the comparison:
   - The final box matches the model's grid cells, so the model decided the
     boundary. A wrong split or merge is a vision call. Look at the prompt and
     the gridded image the model saw.
   - The final box is wider or shifted, so the geometry chain changed it. Start
     at `tighten_region`. Its non-shared-rows branch takes the full word extent
     in the vertical band, which pulls in an adjacent prose column that shares
     the band.

## To find a run's traces

Tracing is gated on `TRACE_TO_LANGSMITH` plus `CC_LANGSMITH_API_KEY`
(`agents/langsmith_tracer.py`). When on, every agent call posts a run.

1. Pick the project by where the run executed:
   - Local runs land in project `default` (the local `.env` sets
     `CC_LANGSMITH_PROJECT=default`).
   - Cloud ECS runs land in project `quberai` (set in
     `deploy/terraform/ecs.tf`).
2. Filter by span name:
   - `grid_locate` — the locator. Its `outputs.messages[0].content` is the full
     `GridFlagResult` JSON, the per-table grid spans the model reported.
   - `vet_structure` — the grounded structure correction.
3. Resolve the project, then query its runs. The key is `CC_LANGSMITH_API_KEY`:

```bash
SID=$(curl -s "https://api.smith.langchain.com/api/v1/sessions?name=default" \
  -H "x-api-key: $CC_LANGSMITH_API_KEY" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)[0]['id'])")
curl -s -X POST "https://api.smith.langchain.com/api/v1/runs/query" \
  -H "x-api-key: $CC_LANGSMITH_API_KEY" -H "Content-Type: application/json" \
  -d "{\"session\":[\"$SID\"],\"filter\":\"eq(name, \\\"grid_locate\\\")\",\"order\":\"desc\"}"
```

## To read how a table was produced

Each table in the output JSON records its provenance. Read these to attribute
behavior to a stage:

- `som_region` — the locator's region (vision).
- `bbox` — Camelot's detected box inside that region.
- `flavor`, `camelot_accuracy` — the Camelot pass.
- `llm_corrected` — True when the structure correction was accepted and its
  markdown differs from the grid render. False covers both a correction that
  was skipped or rejected and an accepted one that left the markdown
  unchanged. An accepted correction leaves a `status` on every non-empty
  `corrected_grid` cell; a skipped or rejected one leaves them all empty.

## To confirm you are looking at the right artifact

A run writes to the path in its log line. Check the file's mtime against the run
you are diagnosing before trusting it, since an older artifact from a prior
version is easy to mistake for the current one.

## Known boundary behaviors

This is an understood limitation, not a new bug. Recognize it before opening
work, and treat it as the baseline.

**Shared-header stacked tables lose column identity.** When several sub-tables
stack vertically under a single shared column-header band, the locator regions
each sub-table separately, so each one's cropped image starts *below* the shared
header. The structure-correction step sees no header in the crop and falls back
to generic labels (Column 1, Column 2, ...). The values are correct and aligned;
what is missing is which period or category each column belongs to. A prompt
change cannot fix this, because the header is physically absent from the crop the
model is given. The intended fix is to carry the shared header down as column
metadata onto the sibling sub-tables, rather than expecting the model to recover
it from a crop that does not contain it.