# Consolidate the Docling spine with Camelot/SoM tables (and stop dropping Docling output)

_Jira: QUE-261_

## Context & Goal

**Goal:** one unified `DoclingDocument` where the spine stays Docling (reading order, headings, text, page structure) but **table bodies and boundaries come from the Set-of-Mark (SoM) / Camelot extraction**, which is text-layer-grounded and vetted.

**Two blockers, established this cycle (`experiments/que259/`):**

1. **quber throws away most of Docling's output.** `docling_parser.py:124-125` does `result = converter.convert(...)` then `return result.document` — discarding `result.confidence` (per-page parse/layout/table/ocr scores) and the parsed-page OCR provenance (`TextCell.from_ocr` + `confidence`). Without that we cannot classify a region as OCR/image or reconcile it.
2. **Nothing reconciles the two extractions** — matching them by count and box, moving Camelot bodies into the spine, and using Docling's structure and classification to correct the SoM tables in return.

**The approach.** Docling and SoM/Camelot both extract the same page, and each is stronger at different things, so the plan lets them enrich each other. SoM/Camelot produces the vetted, text-grounded table bodies. Docling produces the spine, which is reading order and structure, plus an independent table count, picture classification, a text-layer signal, and the OCR cell data for regions read from the page image. That OCR data is the piece quber dropped until now, and capturing it completes a comprehensive view of each page we did not have before. Camelot's table bodies move into the DoclingDocument. Docling's signals correct the SoM/Camelot tables in return: a region SoM merged is split, a chart SoM mistook for a table is annotated, and a table that is really an image is annotated. The plan produces two corrected outputs, the SoM/Camelot tables and the unified DoclingDocument. The comparison that drives all of it is table count plus bounding-box overlap.

## Phase 1 — Expected Docling output (stop dropping data)

The foundational deliverable. State the **output contract** Docling must hand us, then change the parser to produce it.

### What we capture (today: only the first line)

| Source | Field | Carries |
|---|---|---|
| `result.document` | DoclingDocument | tables (prov bbox, `data.table_cells`, num_rows/cols), pictures (prov bbox, `annotations.predicted_classes`), texts, groups, pages |
| `result.confidence` | ConfidenceReport | document + **per-page** `parse_score`, `layout_score`, `table_score`, `ocr_score`, `QualityGrade` |
| `result.pages[i].parsed_page` | list[TextCell] | per cell: `text`, `rect` (TOPLEFT pts), **`from_ocr`**, **`confidence`** — gated behind `generate_parsed_pages=True` |
| derived (computed once) | per-TableItem | `from_ocr` fraction + mean OCR confidence of cells under the table bbox; native-vs-OCR verdict |

### Why it is currently empty

`generate_parsed_pages=False` by default, so Docling **frees** `parsed_page` after assembling the document (verified: `parsed_page is None`, `cells == 0` post-convert). And quber never reads `result.confidence`.

### Verified signal (one scanned doc vs one native doc)

```
scanned page : parse_score=NaN  ocr_score=0.98  from_ocr_frac=1.00   (Camelot empty, acc 0.0)
native page  : parse_score real  from_ocr_frac=0.00                  (Camelot acc 97.7, clean values)
```

The per-region diagnostic (`experiments/que259/diagnostic.py`) already pairs these: OCR-ran vs text-layer, docling reconstruction vs Camelot body.

### Parser change

- Set `generate_parsed_pages=True` in `build_pipeline_options()`.
- Stop returning only `result.document`. The `Parser.parse` contract (`core/parsers/base.py:16`) currently returns `DoclingDocument`; change it to return a richer result object (document + confidence + parsed-page cells + derived per-table provenance). **This is a Protocol change** — it touches `mock.py` and every `parse()` consumer; wire it to a real call site in the same change.

### Serialized output

Three files written alongside the source: `*.docling.json` (the document), `*.confidence.json` (the per-page scores), and `*.cells.json` (the parsed-page cells with `from_ocr` and confidence).

## Phase 2 — Reconciliation model (match tables by overlap)

Work one page at a time. There are only a handful of tables per page, so we match every SoM/Camelot table against every Docling table by bounding-box overlap. No spatial grouping needed. The cardinality of each match is the signal.

- **Matcher:** overlap each SoM table against each Docling item, reusing `coverage_fraction` from `core/extractors/camelot/correspondence/geometry.py`. Normalize boxes to one frame first — Docling prov bbox is BOTTOMLEFT points, SoM `som_region` is normalized TOPLEFT, Camelot `bbox` is BOTTOMLEFT points.
- **Match threshold:** the one knob, default `MATCH_FRACTION=0.20`.
- Each SoM table carries its captured signals — the `from_ocr` fraction and any overlapping Docling picture class — used by the chart and image rows below.

### What each match means

Match each SoM/Camelot table to Docling by overlap, then act on what it matched. Both outputs are corrected in the same pass:

| SoM table matches | meaning | DoclingDocument | SoM/Camelot tables |
|---|---|---|---|
| a Docling table, counts agree | both agree | replace the Docling body with the Camelot body | unchanged |
| a Docling table, Docling has more | SoM merged stacked tables | graft the split tables | `split_table` replaces the merged table with the splits |
| a Docling table, Camelot has more | Docling dropped or merged | split and amend the Docling object up to Camelot | unchanged |
| a Docling table, no text layer under it | a real table rendered as an image | keep Docling's OCR'd table | annotate as an image table |
| a Docling **picture**, not a table | SoM mistook an image for a table | nothing — Docling already holds the picture | annotate as a chart |
| nothing in Docling | Docling missed the table | insert the Camelot table | unchanged |
| a Docling table that no SoM table matched | SoM/Camelot missed a table Docling found | raise an error, no change | untouched — never synthesize Camelot data from Docling |

The picture row and the no-text-layer row lean on Docling's classification and OCR signal, both already in place. The mechanics for `split_table`, the graft calls, and the annotations follow.

## Phase 3 — Graft & reconcile up to Camelot (C >= D)

docling-core exposes the mutation API natively — `insert_table`, `replace_item`, `insert_item_after_sibling`/`_before_sibling`, `delete_items`, `add_table_cell` — so grafting needs **no custom serializer**.

| Case | Action | docling-core call |
|---|---|---|
| **D == 0, C >= 1** (Docling miss) | insert the Camelot table(s) at the right reading-order position | `insert_item_after_sibling` near the bbox-preceding spine element |
| **D == C** (matched) | replace each Docling table body with its Camelot body, spine untouched | `replace_item` (or rebuild `TableItem.data` from Camelot cells) |
| **C > D >= 1** (Camelot finds more — Docling dropped/merged a complex table) | split & amend the Docling object into C tables, graft the Camelot bodies | `delete_items` + C × `insert_table` |

The last row generalizes the old "Docling 1 / Camelot 2" to **any C > D** — Camelot's finer count is trusted because Docling drops/merges complex tables.

The Camelot body is authoritative on cells and boundaries (no completeness gate — SoM is already vetted). The spine (headings, reading order, surrounding text) must be verifiably unchanged.

**Immutability:** clone the DoclingDocument and mutate the clone, keeping the source IR pure (preferred per constraints) — rather than mutating in place.

## Phase 4 — Docling finds more than Camelot: call split_table

When Docling finds **more** tables than Camelot in one region (D > C), the cause is SoM having merged stacked tables into one. The fix is to call `split_table`, a function that already exists at `core/extractors/set_of_mark/split.py:51` but is not yet called from the pipeline.

`split_table(table, count, boundaries) -> list[ExtractedTable]`:
- **Input:** the captured SoM table for the region, Docling's table count for that region, and Docling's per-table boundaries.
- **Output:** a list of tables. They replace the merged table in the SoM output and are grafted into the DoclingDocument.

The work here is the wiring: gather Docling's count and boxes for the region, pass them to `split_table`, and use what it returns in both outputs. The canonical case is `VISA_991_Q126` page 11, the stacked 2025 / 2024 reconciliations SoM fused into one. The two methods disagreeing here is expected, not a defect — SoM and Docling are different probabilistic extractors, and exploiting where they diverge is the whole point of the swap.

## Phase 5 — Annotate charts and image tables on the SoM output

Two SoM tables are not real text tables. Neither is dropped — each is annotated on the `ExtractedTable`, using signals Docling already produces.

- **Chart mistaken for a table** — the SoM table matches a Docling **picture** (Docling's `predicted_classes` reads `bar_chart` / `line_chart` / etc.) and no Docling table. This is the consistent Camelot misclassification that Docling catches. Annotate the `ExtractedTable` as a chart. It stays in the output, marked, but out of the clean table set, and it is not grafted into the DoclingDocument as a table.
- **Table rendered as an image** — the region has no text layer (`from_ocr` ~1.0, Camelot empty) on a Docling-detected table. Annotate the `ExtractedTable` as an image table. Docling's OCR'd cells give this table content for the first time, and that OCR'd table stays in the DoclingDocument. Cleaning the OCR text into final values is a separate step.

The annotation is a **new field on `ExtractedTable`** recording the kind: text table, chart, or image table. The model has no such field today, so this adds one. Docling's classification is used as-is — nothing new is built on the classification side.

## Graft mechanism

**Decided:** graft with docling-core's native mutation API — `replace_item`, `insert_table`, `insert_item_after_sibling` — not a custom serializer, since those calls already exist.

The reconciliation produces a **new DoclingDocument** rather than mutating the source, so the input stays immutable.

Reading-order placement: insert each Docling-missed table right after the spine element directly above it on the page, using `insert_item_after_sibling`. For a table at the top of a page, insert before that page's first element. The one thing to validate on a real document is that the table lands between the correct neighbors.

## Build sequence

1. **Phase 1 — Docling output capture.** Parser change, output contract, and derived per-table provenance. Independently shippable, and a candidate to carve into a sub-task if it grows.
2. **Phase 2 — Overlap matcher.** Match SoM tables to Docling tables by bbox; record each match's cardinality and the picture/text-layer signals.
3. **Phase 3 — Graft actions for C >= D.** insert, replace, or split-amend via docling-core mutation.
4. **Phase 4 — Call split_table for D > C.** pass Docling's count and boxes, graft what it returns.
5. **Phase 5 — Detect, flag, and surface image and chart regions.** tabular-image and chart-as-table.

Evidence per phase on real financial PDFs (Visa Operational Performance, `VISA_991_Q126` p11, plus the scanned daloopa docs in `experiments/que259/inputs/`).

## Scope boundaries

**In scope:** capture Docling's full output, match the two extractions by count and bbox, move Camelot bodies into the DoclingDocument, split merged SoM regions with `split_table`, amend Docling where it under-counts, and annotate charts and image tables on the SoM output. The plan produces two corrected outputs: the SoM/Camelot tables and the unified DoclingDocument.

**Out of scope (named, not dropped):**
- **Tabular-OCR cleanup** — turning a scanned table's OCR text into clean values. Separate post-process refinement. A known approach exists; not this ticket.
- **Cross-page table spanning** — parked. Docling's spanning behavior is unknown.

## Orchestration boundaries (local now, cloud-ready)

The operation is **fusion**, not reconciliation or a join: two independent
extractions, each stronger at different things, share signal so both outputs
improve. We ship a **local** implementation this ticket, but draw the module
boundaries so cloud orchestration is a later wiring job, not a refactor.

Three modular steps, each with a **serializable artifact contract** as its seam:

| Step | Input | Output artifact | Cloud host (later) |
|---|---|---|---|
| document | source PDF | `ParseResult` -> `*.{docling,confidence,cells}.json` | RunPod (GPU) |
| table | source PDF | `list[ExtractedTable]` -> `*.tables.json` | Fargate (CPU) |
| fusion | the two artifacts + source PDF | `FusionResult` -> `*.unified.{json,md}`, `*.tables.json`, `*.fusion.json` | Fargate (CPU) |

The fusion step is CPU + Haiku-API only (the `split_table` path re-runs Camelot
and the LLM); it never needs a GPU. Its core is a free function
`fuse_artifacts(parse, som_tables, source, llm) -> FusionResult` that takes
already-produced data and knows nothing about how it was produced. That function
is the unit a cloud job runs.

Two control flows over the same core:

- **Local (this ticket):** `DocumentFusion.fuse(source)` runs document + table
  in-process, then `fuse_artifacts(...)`. `quber fuse <pdf>`.
- **Cloud (later, separate work):** `quber fuse <pdf> --artifacts-dir <prefix>`
  loads the serialized document + table artifacts from the prefix and runs only
  `fuse_artifacts(...)`. `resolve_document`/`output_sink` already handle `s3://`,
  so the entrypoint works unchanged in cloud.

**Deferred to a cloud ticket (not built now):** the fan-out/fan-in coordinator.
Decided shape: stateless one-shot jobs, no broker. A DynamoDB rendezvous item per
`job_id` records each leg's completion with an atomic conditional `UpdateItem`;
the worker that completes the second leg launches the stateless fusion task
(`ecs.run_task`). No Celery (a broker to operate, fragile chord callbacks over
ephemeral spot GPUs) and no DAG engine for a three-node graph.

## Decisions

- **Parser output** — `parse` returns a `ParseResult` (document + confidence + parsed-page cells + per-table provenance). Blast radius is two callers, `cli.py` and `table_inference.py`, each a one-line change to read `.document`.
- **Match coverage threshold** — keep the existing `0.20` default.
- **Reading-order insertion** — anchor each inserted Docling-missed table to the nearest spine element directly above it on the page and insert right after it with `insert_item_after_sibling`. For a table at the top of a page, insert before that page's first element. Validate on a real document.
- **Name** — the step is **fusion** (`quber fuse`, `core/fusion/`, `fuse_artifacts`, `DocumentFusion`), chosen over reconcile/join/converge: it is multi-source data fusion, neutral on agreement, so it also fits the divergences we surface as errors.
- **ParseResult round-trip** — `ParseResult.save(dir, base)` / `ParseResult.load(dir, base)` make the document step's output a real artifact the fusion step can reload, the seam the cloud handoff rides on.