joSrSSKJr SSKrSSKJr SSKJrJrJ r J r J r J r J r SSKJr SSKJrJr SSKJr \R*"S \R,5rS rS rS rS rSr"SS\5r"SS\5rS*SjrS+Sjr S,Sjr!S-Sjr"S.Sjr#S/Sjr$S0Sjr%S1Sjr&S2Sjr'S3Sjr(S4Sjr)S5Sjr*\4S6Sjjr+S7S jr,S8S9S!jjr-\ "S"S#\ 55r."S$S%5r/"S&S'5r0\S(r1S:S;S)jjr2g)Label of the row this figure was placed into; set by round_off row_labelN)__name__ __module__ __qualname____firstlineno____doc__rr__annotations__rrrrr!__static_attributes__r"7/home/mande/repo/quber/src/quber/agents/completeness.pyrr`s #abE3b!?@Au@!PQAuQ%*K&D " aFC `Isr*rct\rSrSr%\"SS9rS\S'\"SSS9rS \S '\"SS S9rS \S '\"\ S S9r S\S'Sr g)CompletenessVerdicttuTrue if every value-like figure adjacent to the extracted data (in its column span) is present in the extraction. False if a formatted figure sits just past the top or bottom edge but is absent from the extraction — the table was cut off at that edge.rboolcompleter zWhen complete is False, which edge was truncated and an example of the missing figure(s) (e.g. 'the table continues below the last extracted row (e.g. 240,083)'). Empty when complete.rrgapz:One short sentence summarizing the value-token comparison.reasonzfFigures found past an edge but absent from the extraction; the fill step's input. Empty when complete.)default_factoryrzList[MissingFigure]missing_figuresr"N) r#r$r%r&rr0r(r1r2listr4r)r"r*r+r-r-tsf .Hd 5CPFC,1|,O(r*r-c2[R"SSU5$)zCollapse a value token to a comparable key: drop currency, commas, parentheses and surrounding whitespace; keep digits, decimal point and percent. So '$240,083', '240,083' and '(240,083)' all key to '240083'. z[\s,$()]r )resub)tokens r+normalize_valuer:s 66+r5 ))r*c[RU5Vs/sHn[URS55PM sn$s snf)z=Every value-like figure in `text`, normalized for comparison.r)VALUE_TOKEN_REfinditerr:group)textms r+ value_tokensrAs41?1H1H1N O1NAOAGGAJ '1N OO Os$?c^SSKnUR[U55nX1S- nURSSS9Vs/sHBn[US5R 5(dM&[US5R 5PMD snUR 5 $s snf!UR 5 f=f)aThe PDF text layer of one 1-indexed page as its text blocks in reading order, via PyMuPDF. A block is what the layout keeps together: a paragraph, a multi-line title, one panel of a slide. Where the page prints columns, a line-by-line reading interleaves them; blocks do not.rNblocksT)sort)fitzopenrget_textstripclose)sourcepagerGdocpgbs r+ page_blocksrQs  ))CK C ]+-;;xd;+Ka+KasSTUVSWyO`!AaD !+Ka b s"B%B" BBBB,c SSKnUR[U55nX1S- nURnUR S5Vs/sHFn[ US5[ US5[ US5[ US5[US54PMH nn[ UR 5[ UR5U4UR5 $s snf!UR5 f=f)aRead the PDF text layer for one 1-indexed page via PyMuPDF. Returns the page width/height in points and the word boxes as (x0, y0, x1, y1, text) in PyMuPDF's top-left-origin point frame. This is the same text layer Camelot parses, so the figures match exactly. rNrCwordsrF) rGrHrrectrIrwidthheightrK)rLrMrGrNrOrVwrSs r+ page_wordsrZs ))CK C ]wwUWU`U`ahUi? UiPQU1Q4[%!+uQqT{E!A$KQqT KUi ? TZZ % "4e; ? s&CA C,CCC'cx/n/n/n/nUH{nUcMUupxpUR[Xy55 UR[Xy55 URU[X5- 5 URU[X5- 5 M} U(dg[U5[U5[U5[U54$)zUnion of the matched Camelot chunk bboxes, converted to PyMuPDF's top-left point frame: (x0, top, x1, bottom). Camelot bboxes are PDF points with a bottom-left origin, so y flips against the page height. Returns None if no chunk carried geometry. N)appendminmax) extracted_bboxespage_hxs0xs1topsbottomsbbx1y1x2y2s r+extracted_span_ptsrjsCCDG :  3r; 3r; FS[()vB +,  Hc$iS3w< 88r*c 8[[U55nUbU[USUS5U-n[USUS5U-n[USUS5U-n [USUS5U-n O SUSU4upxp[ XA5n U b"U S[ - n U S[ -n U SU SpOXxpS=p/n/n/n/nUGHunnnnnUU-S- nUU-S- nUUs=::aU::aO OU Us=::aU ::dO M;U Us=::aU ::dO ML[ RU5HnURS5n[U5nUU;aM'UbUcURU5 M@[UU- 5n[UU- 5nU[:a U[:aMrUU::aSOS n U S:XaUOURU5 UR[UUUU S 95 M GM U(dU(dU(d [S S S 9$SSjn!/n"U(aU"RSU!"U5S35 U(aU"RSU!"U5S35 U(aU"RSU!"U5S35 [U5[U5-[U5-n#[SSR!U"5U#S3US9$)aDeterministic edge-truncation check by text-layer value comparison. `region_bbox` is the detector's neighbor-bounded box (normalized 0..1, top-left). `extracted_bboxes` are the matched Camelot chunk bboxes (PDF points, bottom-left). All geometry is reduced to PyMuPDF's top-left point frame. A value-like figure is considered when it sits inside the region and within the extracted chunks' x-span plus COLUMN_PAD_PTS, and is absent from `assembled_markdown`. It is reported only if it sits within EDGE_BAND_PTS of the extracted span's top or bottom edge, and is classified top or bottom by the nearer edge. Figures deeper inside the span or farther past an edge are ignored. When no chunk carries a bbox, every missing figure in the region's own column span is reported with the edge unclear. NrrTrCrU@topbottom)rrrrTzQall value-like figures in the table's column region are present in the extractionr0r2cJSR[RU55$)Nz, )joindictfromkeys)valss r+sample$completeness_verdict..sample*syyt,--r*z5rows above the first extracted row are missing (e.g. )z7the table continues below the last extracted row (e.g. zhvalue-like figures in the region are absent from the extraction, edge unclear (no chunk geometry) (e.g. Fz; zH value-like figure(s) in the table region are absent from the extraction)r0r1r2r4)ru List[str]returnr)setrAr]r^rjCOLUMN_PAD_PTSr<r=r>r:r\abs EDGE_BAND_PTSrr-lenrr)$page_wr`rS region_bboxr_assembled_markdown md_valuesrx0rx1r_topr_botspancol_x0col_x1ext_topext_bot top_missingbottom_missingspan_unknown_missingfiguresx0y0rfrgr?cxcyr@origvd_topd_botrrvpartstotals$ r+completeness_verdictrs,L!345I+a.+a.1F:+a.+a.1F:KNKN3f<KNKN3f<!$fc6!9% . 7D a>)a>)7DG  K "N&(#%G %BB2g_2g_r S Ub%9E%9 "&& ((.A771:D%AI~'/$++A.W %EW %E}$)>6;unU(D EM[~ E Ea H NN=trRdK L!/!&2 >-A"f  .E LVT_M`Laabcd NvVdOeNffghi ))/0D)E(Fa I   s>2 2S9M5N NE  IIe `a  r*c4^[U4SjU55nU(d/$US//nUSSH=nXCSS- [:aURU/5 M)USRU5 M? UVs/sHn[U5[ U5- PM sn$s snf)zNumeric-column center x's (PDF points) inferred from the value-like tokens inside the captured span. The fill step slots a restored row's figures into the column whose center is nearest each figure's x. c3># UHJupp4nTSX$-S- s=::a TS::dM!O M%[RU5(dMAX-S- v ML g7f)rCrmrUNr<search).0rrrfrgr?rs r+ infer_columns..Hs]&+ "RRT 7rw#o 0a 0  0 5C5J5J45P C&+s AAAArrCNr)sortedCOLUMN_GAP_PTSr\sumr)rSrxsclustersrcs ` r+ infer_columnsr@s &+ B  $&qE7)H V |B . 0 OOQC RL   "  &. .XCFSVOX .. .s1!Bc8/nUHrupEpgnXW-S- n [X- 5[:d XF-S- U:aM0[RU5(dUR 5S;aM`UR XH45 Mt SR S[U555$)zLabel for a restored row: the non-figure words at row `y` that sit left of the first numeric column, in reading order. Lone currency and paren glyphs are dropped so the label is just the row's text. rm>r $(rx c3*# UH upUv M g7fNr")r_ts r+rrow_label_at..is0-$!A-s)r} ROW_TOL_PTSr<rrJr\rrr) rSr first_col_xrrrrfrgr?rs r+ row_label_atrXs&(E %g_ rv; $C;(F    & &$**,:M*M  bZ !& 880&-0 00r*c ^^[X5mT(aU(dU/4$/n[USS9HrmU(a?[USSTR- 5[::aUSSR T5 MIUR TRTR T/45 Mt /n/n/nUHupn [XTS[- 5n S/[T5-n U HTm[[[T55UU4SjS9n TRX'U S-Tl U TlUR T5 MV S S RU /U -5-S -nU S :XaUOUR X45 M UR!5n[U5HunnUR U5 M [US S9HunnUR#SU5 M SRU5U4$)aRebuild truncated edge rows from the text layer and merge them into the markdown. Returns (new_markdown, placed_figures). Deterministic and value-preserving: the inserted figures are the literal text-layer tokens (the same source Camelot reads), never invented. Each placed figure has its column index and row label set so the caller can record the amendment. A bottom-edge row is appended; a top-edge row is inserted at the top. Returns the markdown unchanged with an empty list if no columns can be inferred or there is nothing to place. cUR$rrr@s r+round_off..ACCr*keyrrrTr c:>[TUTR- 5$r)r}r)icolsmfs r+rrsSa2445Hr*rCz| z | z |rnT)reverse )rrr}rrr\rrrrr]rangerrr!rr splitlinesinsert)rrSrr4rowsplaced top_lines bottom_linesrrfigslabelcellscirow_mdlinesrrrs @@r+ round_offrls  %D !2%%9;D_-8 CR bdd*+{: HQK  r " KKrww- . 9 #%F)+I,.L UtAw'?@s4y BU3t9%+HIBEIQBI BL MM"    E7U?33d:em==qkJ  ) ) +EL) 6 V*It4 6 Q5 99U V ##r*c^U(d/$[SU55n/n[U5H3m[U4SjUSS55(dM"URT5 M5 U$)zGrid column indices that hold value-like figures in the data rows (row 0 is treated as the header). Ordered left to right; these are the columns a restored row's figures get slotted into. c38# UHn[U5v M g7frrrrs r+r'numeric_grid_columns..s&1Ac3~># UH2nT[U5:=(a [RUT5v M4 g7fr)rr<r)rrowjs r+rrs0S#q3s8|= 5 5c!f ==s:=rCN)r^ranyr\)rrWoutrs @r+numeric_grid_columnsrsY  && &EC 5\ SqrS S S JJqM Jr*cj[R"SSU5R5R5$)zdCollapse whitespace and case for matching a restored row to an existing grid row by its label cell. z\s+r)r7r8rJlower)r?s r+normalize_labelrs) 66&#t $ * * , 2 2 44r*c "^^^^[X5m[U5mT(aT(aU(dUVs/sHn[U5PM sn/4$UVs/sHn[U5PM nn[SU5SS9nUH$mTR S/U[ T5- -5 M& SUU4Sjjn/n[ USS9Hrn U(a?[USSU R- 5[::aUSS RU 5 MIURU RU RU /45 Mt /n UGHupn [XSRTS[- 5nU V s0sH#n [U"U R5US - 5U _M% nn S nU(a[[!U5HLunm[#U4S jT55nT(dM$[%TS5[%U5:XdMAU(dMJUn O UcMS/[US 5-nUUS'U S :Xa [ U5O[S [ U55nUR'UU5 UR)5Humn UUTR+5(aNTVs/sH*nUU:dM UUUR+5(aM(UPM, nnU(dMa[UU4SjS9mU R,UUT'TU lXlU RU 5 M GM XZ4$s snfs snfs sn fs snf)aRound off a truncated table by inserting the missing edge figures into a structured cell grid (not a markdown string). Returns (new_cells, placed_figures). Clean composition: a figure is placed into the grid column whose inferred x matches the figure's x (text-layer numeric columns map left-to-right onto the grid's numeric columns). A restored row is filled in place into the first grid row whose label cell matches and which has at least one empty numeric column, so the column count is preserved. Otherwise a new row is inserted, at the bottom for a bottom-edge truncation and after the header for a top-edge one. A figure whose mapped cell is already full goes to the nearest empty numeric column of that row. If none is empty the figure is left unplaced and the re-audit still reports it. Values are the literal text-layer tokens, never invented; a non-empty Camelot cell is never overwritten. c38# UHn[U5v M g7frrrs r+r!round_off_grid..s+AQrr)rr cz>^[[[T55UU4SjS9nU[T5:aTU$TS$)Nc&>[TUT- 5$rr})rcol_xrs r+r8round_off_grid..grid_col_for_x..sU1X\1Br*rr)r]rr)rkrnum_colss` r+grid_col_for_x&round_off_grid..grid_col_for_xs7 c%j!'B C#h-/x{AXb\Ar*cUR$rrrs r+r round_off_grid..rr*rrrTrCNc3|># UH1o[T5:=(a TUR5(+v M3 g7fr)rrJ)rrrs r+rrs,Q13q6z>!A$**,.>>s9<roc >[UT- 5$rr)rgcs r+rrs AF r*)rrrzr)rrr5r^extendrrr}rrr\rrrr]r enumeraterrritemsrJrrr!)rrSrr4r new_cellsrWrrrrrrrrwanttargetrihas_roomrremptiesrrrs ` @@@r+round_off_gridrs, % &E#E*H !&'AQ'++38!95a$q'5I!9 ++Q 7E  "Q()BB :$UA&CCF'+x'7S^SC N=SF   VS )jjlFB $**,,'/ah!e)1IfDUVWDXD^D^D`1ha&;<$&HHIf b !BI L MM" #3L  u(!9,I2bs#K=L"*L7 L L "L c[SU55n/nSnUH7upVUbXT- U:aUR/5 UnUSRU5 M9 U$)zEGroup words into visual rows by y-center (words within `tol` points).c3@# UHoSUS-S- U4v M g7f)rCrUrmNr"rrYs r+rtext_rows..s$;UqTAaD[C'+UsNr)rr\)rStolrrrow_cyrrYs r+ text_rowsr s_ ;U; ;E?AD"F >R[3. KKOF R  Kr*c :/n[U5Hn[SU55S:aMUVs/sHo3SPM snUVs/sHo3SPM sn-nUVs/sHo3SPM snUVs/sHo3SPM sn-nUR[U5[ U5[U5[ U545 M U(d/$UR 5 [ US5/nUSSHuupxpXvSS- [::aJ[ USSU5USS'[USSU 5USS'[ USSU 5USS'MbURXxX/5 Mw UV s/sHoSU SU SU S4PM sn $s snfs snfs snfs snfs sn f)aVertical bands of contiguous *tabular* rows, each (top, bottom, x_left, x_right) in points. A tabular row carries two-or-more value-like figures (data rows of a real table); prose lines and lone labels are excluded. Contiguous tabular rows (vertical gap <= BAND_GAP_PTS) merge into one band. Used to relocate a detector box onto the text that is actually there, instead of trusting a box that landed in whitespace. c3d# UH&n[RUS5(dM"Sv M( g7f)rFrCNrrs r+r tabular_bands..&s#=#Q!6!6qt!.Ysqtr*r)r]r^)box_normr rr` near_pad_fracbtrebestbest_ovbandovpadnearr1rnr r rs r+ repair_boxr9s$ Xa[(1+ & /B Xa[(1+ & /B8rPs%N# XXX%' JJ .    I()6*P *9K9 9194c c c <c= c L c  ccL/ ;/ +//01 ;1 11 1(/$/$ ;/$ ,/$- /$ % /$d 5S "S ;S ,S- S 1 Sp  ;   8  5 ;5,5H "B/"B 6"B "B  "B  "B 1 "BJ "( " "$ $ N&n-_r*