"""Put each figure's text onto the picture the parse already found. The placeholder is already in the document. Every picture the parse detected sits in the body in reading order carrying its page and its box, and nomination read that placeholder rather than creating it. Grafting fills it in: the picture that was always there now carries the figure's content. So a chart is not a new kind of element competing with tables for a position. It carries no cells and no per-value location, and is not shaped like a table. The text lands as the picture's description, the document model's own field for what a picture depicts, and the scan's identity lands in the picture's metadata so a picture can be traced back to the scan that read it and a model change is visible without re-reading the text. Each record is given the reference of the picture it filled, so a plotted value traces back to the element that holds it. The two counts do not always agree. The scan decides, because it looked at the page and the box it drew is already a source coordinate, while the parse is working from a layout model. So a region ends up holding exactly as many pictures as the scan returned figures: - One picture the scan read as several figures is split into one per figure. The parse draws a single region over two charts printed side by side. - Several pictures the scan read as one figure become one. The parse splits a chart pair the scan describes together. - A region already agreeing keeps the box and bitmap the parse measured. - A figure over no picture at all is surfaced as an error and never dropped, since a figure the scan read that nothing holds is a value that vanishes. Figures and pictures are grouped by overlap before anything is assigned, so the order the figures arrive in never decides an outcome. What a picture *is* remains the parse's call, not the scan's. The scan describes a wordmark as readily as a chart, and a branded deck prints one on every page, so a region the parse classified as page furniture is dropped with its records. That judgement reads a class the parse already published rather than a rule written against a description that has no fixed shape. The text the parse lifted out of a picture is removed once the scan has read it, except for items whose label the caller names in `preserve`. Those fragments are bare numbers with nothing saying what they measure, and the description now carries the same values with their series and their axes. Keeping both would put two readings of one chart in the document. This module attaches no titles or notes to a picture. One note commonly serves several charts on a page, and nothing here matches a superscript to the note it points at. The correction sweep in `quber.core.figures.correct` writes each figure's markers and page notes onto its picture's metadata under `FOOTNOTE_MARKS_FIELD` and `FOOTNOTES_FIELD`. On the dpt-3 track, `quber.core.figures.dpt3.footnotes` then resolves each marker to its note. A table read off a page image goes into the table the parse already found for it, the same way. The element stays a table; only its body changes, from the parse's own reading of the image to the scan's, along with the title, subtitle, units and footnotes read off the page with it. `graft_figures` never mutates the document it is given and returns a clone. `graft_tables` mutates the document it is given. The orchestrator runs it first, on its own copy of the parse, and then hands that copy to `graft_figures`. """ from __future__ import annotations from pathlib import Path from typing import Collection, Dict, List, Optional, Tuple from docling_core.types.doc.base import BoundingBox, CoordOrigin from docling_core.types.doc.common.meta import DescriptionMetaField from docling_core.types.doc.common.reference import ImageRef, ProvenanceItem from docling_core.types.doc.document import DoclingDocument, PictureItem, PictureMeta, TextItem from docling_core.types.doc.labels import DocItemLabel from loguru import logger from quber.core.extractors.base import ExtractedTable from quber.core.extractors.camelot.correspondence.geometry import coverage_fraction from quber.core.figures.geometry import NormBox, norm_box, prov_box from quber.core.figures.models import FigureRecord, PageScan, ScannedTable from quber.core.figures.nominate import FURNITURE_CLASSES, picture_classes from quber.core.fusion.graft import READ_BY_FIELD, attach_attribution, markdown_to_table_data from quber.files.pdf import render_region #: Minimum box overlap for a returned figure and a parse picture to be the same #: region, measured both ways so a large picture containing a small figure and a #: figure drawn wider than the picture both register. Carried over from the #: table matcher, whose regions are the same size and drawn by the same kinds of #: detector. MATCH_FRACTION = 0.20 #: Two figures whose tops sit within this fraction of the page height are printed #: side by side rather than stacked. A chart occupies far more of a page than #: this, so the band separates rows without needing to know the layout. ROW_BAND = 0.05 #: What the picture's description records as its author. DESCRIPTION_AUTHOR = "quber-chart-scan" #: The reader this workflow uses, recorded on every region it reads so a later #: reader can see the region is already read and by what. ADE = "ade" #: Where a figure records the footnote reference markers its labels carry, each #: with what it points at. A table states its markers as an attribute rather than #: leaving them inside its rendered body, and a figure states them the same way, #: so resolving them later reads a field instead of the description they sit in. FOOTNOTE_MARKS_FIELD = "quber__footnote_marks" #: Where a figure records the notes printed on its page, as marker and text. The #: agent reads them off the image the same way a table's correction agent reads #: the notes below the table, so resolution starts from a pair it was given #: rather than from a scan over the page's lines. FOOTNOTES_FIELD = "quber__footnotes" #: Resolution for a re-cut bitmap when the picture being split carried none to #: match. 72 dots per inch is one pixel per PDF point, the document model's own #: unscaled default. DEFAULT_CROP_DPI = 72 def graft_figures( document: DoclingDocument, scans: List[PageScan], page_dims: Optional[Dict[int, Tuple[float, float]]] = None, match_fraction: float = MATCH_FRACTION, source: Optional[Path] = None, preserve: Collection[DocItemLabel] = (), ) -> Tuple[DoclingDocument, List[str]]: """Carry each scan's figure text onto the matching picture in a clone of `document`. `page_dims` gives each page's width and height in points, used to bring the parse's picture boxes into the scan's normalized frame. Pages absent from it fall back to the document's own page sizes. `source` is the document the parse was made from. Splitting a picture moves the boundaries its stored bitmap was cut to, so each part's bitmap is cut again from the page. Without the source a split part carries no bitmap and the loss is logged. `preserve` names text labels that survive under a picture the scan has read. The scan's reading supersedes the parse's stray content fragments, but an item the parse labelled a footnote or a caption is native page text with a known role, not a reading of the figure, and a caller passing its label keeps it in the document. Returns the refined document and one error string per figure that overlaps no picture. The records in `scans` are given the reference of the picture each one filled, and a record whose region the parse called page furniture is removed from the scan. """ refined = document.model_copy(deep=True) dims = dict(page_dims or {}) for page_no, page in refined.pages.items(): if page_no not in dims and page.size is not None: dims[page_no] = (page.size.width, page.size.height) pictures_by_page: Dict[int, List[PictureItem]] = {} for picture in refined.pictures: if picture.prov: pictures_by_page.setdefault(picture.prov[0].page_no, []).append(picture) errors: List[str] = [] # Structure is reshaped first and the text attached afterwards. Removing a # picture renumbers the ones after it, so a reference read before the last # reshape would name the wrong element. pairs: List[Tuple[PictureItem, FigureRecord, PageScan]] = [] for scan in scans: width, height = dims.get(scan.page, (612.0, 792.0)) candidates = pictures_by_page.get(scan.page, []) regions, unmatched = _regions(scan.figures, candidates, width, height, match_fraction) for chart in unmatched: errors.append( f"page {scan.page}: the scan returned a figure at {chart.box} that overlaps no " f"picture in the parse (job {chart.job_id}); its text is in the figure records only" ) for pictures, figures in regions: if _is_furniture(pictures): _discard(scan, figures, pictures) continue pairs.extend( (picture, chart, scan) for picture, chart in _reshape(refined, pictures, figures, scan, width, height, source) ) for picture, chart, scan in pairs: _attach(picture, chart, scan) # Done in one pass at the end: removing a text renumbers the texts after it, # and the pictures were being collected until the last attachment. _discard_superseded(refined, [picture for picture, _c, _s in pairs], preserve) return refined, errors def graft_tables( document: DoclingDocument, scans: List[PageScan], tables: List[ExtractedTable], page_dims: Dict[int, Tuple[float, float]], ) -> List[str]: """Put each table read off a page image into the parse table it was read for. The element stays a table. Nothing is relabelled: a balance sheet is a table whichever tool read it, and calling it an image so it could travel a picture-shaped path would bend the document to suit the code. The extracted body replaces the parse's own reading of the image, and the title, subtitle, units and footnotes the correction read off the page are attached with it, exactly as they are for a table the table engine produced. Nothing is lost by replacing: the page's raw response is stored as it came back, so what was read and by which model version stays on record. Mutates `document` in place — the caller owns the clone — and returns one error per scanned table it could not put in: one whose parse table or extraction cannot be found, one whose extraction has an empty body, and one read over a picture that cannot be resolved or produced no body. """ by_id = {t.table_id: t for t in tables if t.table_id} by_ref = {t.self_ref: t for t in document.tables} errors: List[str] = [] by_picture = {p.self_ref: p for p in document.pictures} for scan in scans: for scanned in scan.tables: if scanned.table_id is None: continue if scanned.table_ref is None and scanned.picture_ref is not None: _add_beside_picture(document, scan, scanned, by_picture, by_id, page_dims, errors) continue if scanned.table_ref is None: continue table_item = by_ref.get(scanned.table_ref) extracted = by_id.get(scanned.table_id) if table_item is None or extracted is None: errors.append( f"page {scan.page}: the table read off the page image ({scanned.table_id}) has " f"no home in the parse ({scanned.table_ref}); it is in the table records only" ) continue if not (extracted.markdown or "").strip(): errors.append( f"page {scan.page}: the scan returned a table over {scanned.table_ref} with no " "cells in it; the parse's own reading of the image is left standing" ) continue logger.info( "Table graft: page {} replacing {} with the scan's reading ({})", scan.page, scanned.table_ref, scanned.table_id, ) table_item.data = markdown_to_table_data(extracted.markdown) attach_attribution(document, table_item, extracted, page_dims) return errors def unread_pictures( document: DoclingDocument, page_dims: Dict[int, Tuple[float, float]], ) -> List[str]: """One error per picture the run left with nothing in it. A figure the scan returned that covers no picture is already reported, so a value the scan read never disappears quietly. The reverse was not: a picture the scan returned nothing over ended the run empty and said nothing about it. That is how a fourteen-row table of property sales went missing on a page the run reported no errors for — the parse had filed the region as a picture, and a picture nothing claimed was indistinguishable from a picture correctly left alone. A picture is accounted for when it carries a description, or when a table in the document shares any area with it on its page. Full coverage is not required. The second case is a region the parse detected twice, once as a table and once as a picture; the table holds the content and the picture is a duplicate outline of it. Page furniture is not reported. A logo is never read on purpose. """ errors: List[str] = [] tables_by_page: Dict[int, List[NormBox]] = {} for table in document.tables: if not table.prov: continue page = table.prov[0].page_no width, height = page_dims.get(page, (612.0, 792.0)) box = prov_box(table, width, height) if box is not None: tables_by_page.setdefault(page, []).append(box) for picture in document.pictures: if not picture.prov: continue classes = picture_classes(picture) if classes and all(c in FURNITURE_CLASSES for c in classes): continue if (picture.meta.description.text if picture.meta and picture.meta.description else "").strip(): continue page = picture.prov[0].page_no width, height = page_dims.get(page, (612.0, 792.0)) box = prov_box(picture, width, height) if box is not None and any(_overlaps(box, other) for other in tables_by_page.get(page, [])): continue errors.append( f"page {page}: picture {picture.self_ref} " f"({', '.join(classes) or 'unclassified'}) was read by nothing — the scan returned no " "figure over it and the document holds no table there; whatever it shows is not in the " "document" ) return errors def _overlaps(one: NormBox, other: NormBox) -> bool: """Do the two boxes share any area at all?""" return min(one[2], other[2]) > max(one[0], other[0]) and min(one[3], other[3]) > max(one[1], other[1]) def _add_beside_picture( document: DoclingDocument, scan: PageScan, scanned: ScannedTable, by_picture: Dict[str, PictureItem], by_id: Dict[str, ExtractedTable], page_dims: Dict[int, Tuple[float, float]], errors: List[str], ) -> None: """Put a table the parse filed as a picture into the document as a table. The parse detected the region and called it a picture. The scan read it and returned a grid, so the page prints a table and the document should hold one. It is added rather than swapped in, because the picture is a real element and the printed image is what the values were read from. Nothing else about it is special. The body is the corrected grid every other scanned table carries, attached the same way, on a provenance box taken from the picture the table was printed over. """ picture = by_picture.get(scanned.picture_ref or "") extracted = by_id.get(scanned.table_id or "") if picture is None or extracted is None or not (extracted.markdown or "").strip(): errors.append( f"page {scan.page}: the scan read a table over picture {scanned.picture_ref}, which the " f"parse holds no table for, and the grid produced no body; the values are in the table " f"records only" ) return logger.info( "Table graft: page {} adding the scan's reading of {} as a table ({})", scan.page, scanned.picture_ref, scanned.table_id, ) table_item = document.add_table( data=markdown_to_table_data(extracted.markdown), prov=picture.prov[0] if picture.prov else None, ) attach_attribution(document, table_item, extracted, page_dims) def _discard_superseded( document: DoclingDocument, pictures: List[PictureItem], preserve: Collection[DocItemLabel] = (), ) -> int: """Remove the text the parse pulled out of a picture the scan has now read. A picture the parse detected carries the fragments it managed to lift off the image as its own children — a chart's plotted labels arrive as `$0.05`, `$0.02`, `$0.06`, bare numbers with nothing saying which series or which year they belong to. Once the scan has read the figure, those fragments are the same region read worse, and keeping them puts two readings of one chart in the document with only one of them saying what the numbers mean. An item whose label is in `preserve` stays: it is native page text with a role the parse named, not a reading of the figure. This is the same replacement a table gets. A table read off a page image has its body replaced rather than doubled, for the same reason and with the same justification: the raw response is on disk, so nothing is unrecoverable. """ doomed = [] for picture in pictures: for child in picture.children: item = child.resolve(document) if isinstance(item, TextItem) and item.label not in preserve: doomed.append(item) if not doomed: return 0 logger.info( "Figure graft: removing {} text fragment(s) the parse lifted from {} picture(s) the scan has read", len(doomed), len(pictures), ) document.delete_items(node_items=doomed) return len(doomed) def _regions( figures: List[FigureRecord], candidates: List[PictureItem], width: float, height: float, match_fraction: float, ) -> Tuple[List[Tuple[List[PictureItem], List[FigureRecord]]], List[FigureRecord]]: """Group figures and pictures that overlap into regions, one region per subject. Every figure is scored against every picture before anything is grouped, so the order the figures arrive in never decides an assignment. Overlap is transitive: a figure joins every picture it overlaps and a picture joins every figure that overlaps it, and the connected result is one region. A region can then hold any mix — one to one, one picture the scan read as several figures, several pictures the scan read as one figure, or a tangle of both. Returns the regions in the pictures' document order, each region's pictures in document order and its figures in reading order, plus the figures that overlap no picture past the threshold. """ boxed = [(p, box) for p, box in ((p, prov_box(p, width, height)) for p in candidates) if box] order = {p.self_ref: i for i, (p, _b) in enumerate(boxed)} # picture ref -> figures over it, and figure index -> pictures under it. over: Dict[str, List[int]] = {} under: Dict[int, List[str]] = {} for i, chart in enumerate(figures): cbox = norm_box(chart.box) if cbox is None: continue for picture, pbox in boxed: cov = max(coverage_fraction(cbox, pbox), coverage_fraction(pbox, cbox)) if cov >= match_fraction: over.setdefault(picture.self_ref, []).append(i) under.setdefault(i, []).append(picture.self_ref) unmatched = [chart for i, chart in enumerate(figures) if i not in under] by_ref = {p.self_ref: p for p, _b in boxed} regions: List[Tuple[List[PictureItem], List[FigureRecord]]] = [] seen_pictures: set[str] = set() seen_charts: set[int] = set() for picture, _box in boxed: if picture.self_ref in seen_pictures or picture.self_ref not in over: continue # Walk the overlap both ways until the region stops growing. refs = {picture.self_ref} idxs: set[int] = set() frontier = [picture.self_ref] while frontier: ref = frontier.pop() for i in over.get(ref, []): if i in idxs: continue idxs.add(i) for other in under.get(i, []): if other not in refs: refs.add(other) frontier.append(other) seen_pictures |= refs seen_charts |= idxs regions.append( ( sorted((by_ref[r] for r in refs), key=lambda p: order[p.self_ref]), _reading_order([figures[i] for i in sorted(idxs)]), ) ) return regions, unmatched def _reading_order(figures: List[FigureRecord]) -> List[FigureRecord]: """Figures in the order the page prints them: down the page, then across.""" def key(chart: FigureRecord) -> Tuple[int, float]: box = norm_box(chart.box) or (0.0, 0.0, 0.0, 0.0) return (round(box[1] / ROW_BAND), box[0]) return sorted(figures, key=key) def _is_furniture(pictures: List[PictureItem]) -> bool: """True when the parse called every picture in the region page furniture. A logo or an icon is the same mark on every page of a deck, and the scan describes it as readily as it describes a chart. The parse's class is the signal for what a picture is, and it is trusted here rather than reading the scan's description to guess — one is a prediction the parse already made and published, the other is a rule written against prose with no fixed shape. """ classes = [c for picture in pictures for c in picture_classes(picture)] return bool(classes) and all(c in FURNITURE_CLASSES for c in classes) def _discard(scan: PageScan, figures: List[FigureRecord], pictures: List[PictureItem]) -> None: """Drop the records for a region the parse called page furniture. The scan read the region and was right about it, so nothing is being corrected. The record is dropped because a description of a wordmark is not document content, and keeping it would put a figure on every page of a branded deck and count it among the figures. """ logger.info( "Figure graft: page {} dropping {} record(s) over page furniture ({})", scan.page, len(figures), ", ".join(sorted({c for p in pictures for c in picture_classes(p)})), ) for chart in figures: if chart in scan.figures: scan.figures.remove(chart) if not scan.figures and scan.status == "figures": scan.status = "empty" def _reshape( document: DoclingDocument, pictures: List[PictureItem], figures: List[FigureRecord], scan: PageScan, width: float, height: float, source: Optional[Path], ) -> List[Tuple[PictureItem, FigureRecord]]: """Make the region hold one picture per figure the scan returned. The scan is the authority on how many figures a region holds and where each one sits. It looked at the page, and the box it draws is already a source coordinate, so the parse's count yields to it in both directions: a picture the scan read as several figures becomes several, and several pictures the scan read as one figure become one. A region already agreeing is returned unchanged. The first picture in document order survives, so the reading-order position the parse gave the region is kept, and any others are removed. Extra figures are inserted behind the survivor. Each resulting picture carries the parse's class prediction for the region. Only a region whose shape actually changed takes the scan's boxes, and its bitmaps are then cut again from those boxes, because a bitmap cut to the old boundary depicts a region that is no longer an element. A region the two sides already agree on keeps the box and bitmap the parse measured — there is nothing to correct, and replacing them would discard a good crop for no gain. Returns the picture paired with its figure, for the caller to attach once every region has been reshaped. """ survivor = pictures[0] template_meta = survivor.meta dpi = survivor.image.dpi if survivor.image is not None else DEFAULT_CROP_DPI wanted_image = any(p.image is not None for p in pictures) reshaped = len(pictures) > 1 or len(figures) > 1 if not reshaped: return [(survivor, figures[0])] if len(pictures) > 1: logger.info( "Figure graft: page {} the scan read {} as {} figure(s); merging into {}", scan.page, ", ".join(p.self_ref for p in pictures), len(figures), survivor.self_ref, ) document.delete_items(node_items=list(pictures[1:])) if len(figures) > len(pictures): logger.info( "Figure graft: page {} picture {} covers {} figures the scan returned separately; " "splitting it into {}", scan.page, survivor.self_ref, len(figures), len(figures), ) survivor.prov = [_prov(figures[0], scan.page, width, height)] survivor.image = _crop(source, scan.page, figures[0], dpi) if wanted_image else None pairs = [(survivor, figures[0])] anchor: PictureItem = survivor for chart in figures[1:]: anchor = document.insert_picture( sibling=anchor, prov=_prov(chart, scan.page, width, height), image=_crop(source, scan.page, chart, dpi) if wanted_image else None, after=True, ) if template_meta is not None: anchor.meta = template_meta.model_copy(deep=True) pairs.append((anchor, chart)) return pairs def _crop(source: Optional[Path], page: int, chart: FigureRecord, dpi: int) -> Optional[ImageRef]: """The part's own bitmap, cut from the page at the resolution the parse used. Returns None when there is no source to cut from or the cut fails, so a split still happens and the part carries a box with no bitmap rather than one that depicts the wrong region. Either way the loss is logged, never silent. """ box = norm_box(chart.box) if source is None or box is None: logger.warning( "Figure graft: page {} figure {} keeps no bitmap ({}); its box is recorded", page, chart.chunk_id, "no source document to cut from" if source is None else "the scan returned no box", ) return None try: return ImageRef.from_pil(image=render_region(source, page, box, dpi), dpi=dpi) except Exception as exc: logger.warning( "Figure graft: page {} figure {} keeps no bitmap; cutting it from the page failed ({}: {})", page, chart.chunk_id, type(exc).__name__, exc, ) return None def _attach(picture: PictureItem, chart: FigureRecord, scan: PageScan) -> None: """Carry the figure's text and the scan's identity onto one picture. Both ride on `meta`. The text goes in its description field, the document model's own place for what a picture depicts, and the scan's identity goes in alongside — the meta model allows extra fields, so they serialize with the document. The picture's existing metadata, its class prediction included, is kept. The record is given this picture's reference in return. """ base = picture.meta or PictureMeta() picture.meta = base.model_copy( update={ "description": DescriptionMetaField(text=chart.text, created_by=DESCRIPTION_AUTHOR), "quber__figure_job_id": scan.job_id, "quber__figure_model": scan.model, "quber__figure_version": scan.version, "quber__figure_chunk_id": chart.chunk_id, READ_BY_FIELD: [*(getattr(base, READ_BY_FIELD, None) or []), {"reader": ADE, "produced": []}], } ) chart.picture_ref = picture.self_ref def _prov(chart: FigureRecord, page: int, width: float, height: float) -> ProvenanceItem: """A provenance record for one figure, its box in bottom-left points.""" x1, y1, x2, y2 = norm_box(chart.box) or (0.0, 0.0, 1.0, 1.0) bbox = BoundingBox( l=x1 * width, r=x2 * width, t=(1.0 - y1) * height, b=(1.0 - y2) * height, coord_origin=CoordOrigin.BOTTOMLEFT, ) return ProvenanceItem(page_no=page, bbox=bbox, charspan=(0, 0))