tjY*:SrSSKJr SSKrSSKJr SSKJr SSK J r SSK J r J r JrJrJrJr SSKJr SS KJrJr \ (aSS KJr \S rS rSS jr"SS\5rSSjrSSjrSSjrSSjr SSjr!SSjr"SSjr#\4S Sjjr$S!Sjr%g)"u Camelot acquisition layer: get candidate cell grids out of a PDF. This module knows nothing about LLMs or the pipeline graph. It renders page images, runs both Camelot flavors in spawn-context subprocesses, and converts each Camelot DataFrame into a serializable `CamelotCandidate`. Both Camelot flavors run in parallel: - `lattice` — visible grid lines. - `stream` — whitespace-based; more permissive. There is deliberately no lattice-first / fall-back-to-stream gate: on pages where tables are aligned by whitespace alone, lattice still returns grids — empty shells — so "fall back only when lattice finds nothing" let those shells through as false positives. `is_content_empty` catches the shells instead. ) annotationsN)ProcessPoolExecutor) TimeoutError)Path) TYPE_CHECKINGAnyListLiteralOptionalTuple)logger) BaseModelField) DataFrame)latticestreamgN@cSSKJn U"[U5US[U5S9n/n[USS9H2upgUSUSS 3- nUR US 5 UR U5 M4 U$) Nr)convert_from_pathpng)dpifmt output_folderstartzpage-04dz.pngPNG) pdf2imagerstr enumeratesaveappend) sourcerout_dirrimagespathsiimgpaths C/home/mande/repo/quber/src/quber/core/extractors/camelot/acquire.py render_pagesr+(sl+ s6{cRYl [FEF!,53t,, u T- Lc\rSrSr%Sr\"SS9rS\S'S\S'\"S S 9rS \S 'S r S\S'\"SSSS9r S\S'\"\ SS9r S\S'\"\ SS9r S\S'\"SS9rS\S'Srg )CamelotCandidate4aSerializable per-Camelot-table output. Crosses the spawn-process boundary; only picklable primitives, no Camelot internals. `cells` is the raw grid (rows of cell strings) straight from Camelot's DataFrame; `markdown` is its rendering. The correspondence flow works on `cells` so it can assemble, audit and fill at the grid level and render markdown once at the end; the legacy classifier/unifier path consumes `markdown`. z1Stable id assigned by the parent; flavor-page-idx) descriptionr candidate_idFlavorflavorr)geintpageNz+Optional[Tuple[float, float, float, float]]bboxgY@)defaultr4lefloataccuracyz)Camelot's raw cell grid (rows of strings))default_factoryr0List[List[str]]cellsa;Per-cell geometry aligned 1:1 with `cells`: same row/column shape, each entry the Camelot cell box (x1, y1, x2, y2) in PDF points with a bottom-left page origin, or None where Camelot exposed no box for that cell. Empty for candidates not built directly from a Camelot grid (e.g. assembled or recovered candidates).7List[List[Optional[Tuple[float, float, float, float]]]] cell_boxesz;Camelot's raw markdown for this table; rendering of `cells`markdown)__name__ __module__ __qualname____firstlineno____doc__rr1__annotations__r6r7r<listr?rArB__static_attributes__rCr,r*r.r.4s*]^L#^ N D#8#grid_to_markdown..Z&1Ac>UVs/sHn[U5PM snS/T[U5- --nSSRU5-S-$s snf)NrM|  |  |)rrQjoin)rowcpaddedwidths r*render grid_to_markdown..render\sN"%&#Q#a&#&"S1A)BBejj((4//'sArrZr[c3&# UHnSv M g7fz---NrCrS_s r*rUrV`s0M 1 r\c34># UH nT"U5v M g7frOrC)rSr^rbs r*rUrVas2  srN )r^ List[str]returnr)maxr]rangeextend)r?linesrbras @@r*grid_to_markdownrqSsz  && &E0E!H tejj0Me 0M&MMPTT UE LL2ab 22 99U r,cSnUS- nU(a4[US- S5up[[S5U-5U-nU(aM4U$)zB0-based column index to a spreadsheet letter: 0->A, 25->Z, 26->AA.rMrA)divmodchrord)indexoutrems r* column_letterr{esJ C QJE EAIr* #c(S.!C' % Jr,c,^^^U(dg[SU55mSSjmSUU4SjjmT"SUS5SSRS [T555-S -/nURU4S j[ USS S S955 SRU5$)uRender the grid with each non-empty cell's address printed inside it. Every cell with text carries an inline tag — `[B3] 1,637` means data column B (A=0), grid row 3 (1-based) — so the agent READS an address off the label sitting next to the text; it never counts rows, columns, or pipes. Blank cells stay blank. The tags are reference only — the agent's corrected output must never contain them. rMc38# UHn[U5v M g7frOrPrRs r*rU-grid_to_addressed_markdown..zrWrXch[U5R5(aS[U5USU3$S$)N[z] rM)rstripr{)rTr_texts r*tag'grid_to_addressed_markdown..tag|s447IOO4E4E=#$QCr$0M2Mr,c>^UVs/sHn[U5PM snS/T[U5- --nSSRUU4Sj[U555-S-$s snf)NrMrZr[c3:># UHupT"TX5v M g7frOrC)rSr_vrTrs r*rUGgrid_to_addressed_markdown..render_addressed..s L:K$!Q:Ksr\)rrQr]r )rTr^r_r`rras` r*render_addressed4grid_to_addressed_markdown..render_addressedsY"%&#Q#a&#&"S1A)BBejj L)F:K LLLtSS'sArrrZr[c3&# UHnSv M g7frerCrfs r*rUr~s=Z\e\rhr\c38># UHupT"X5v M g7frOrC)rSr'r^rs r*rUr~sV8Ufa!!))8UsNrrj)rTr5r_r5rrrlr)rTr5r^rkrlr)rmr]rnror )r?rprrras @@@r*grid_to_addressed_markdownros  && &ENTTaq *D5::=ZUSX\=Z3Z,Z]a,a bE LLV %)ST8UVV 99U r,c URR5VVs/sHoVs/sHn[U5PM snPM snn$s snfs snnf)z5Camelot DataFrame -> raw cell grid (rows of strings).)valuestolistr)dfr^r_s r* df_to_cellsrs?-/YY-=-=-? @-?cS !SSVS !-? @@ ! @s A A A A c 8^ Ub [U5O/n/n[U5HupEU[U5:a [X$5O/n/n[[U55HnU[U5:aXhOSm T b}[ U 4SjS55(acUR [ T R5[ T R5[ T R5[ T R545 MUR S5 M UR U5 M U$)uPer-cell boxes aligned to `grid`'s shape from Camelot's Cell objects. `raw_cells` is Camelot's `table.cells` — rows of `Cell` objects, each with x1/y1/x2/y2 in PDF points (bottom-left origin). `table.df` (hence `grid`) is built from that same grid, so index i,j lines up; still, we clamp to `grid`'s shape and fill None for any position Camelot did not cover, so `cell_boxes` is always exactly the same shape as `cells`. Nc3<># UHn[TU5v M g7frO)hasattr)rSacells r*rU!cells_to_boxes..s'[BZQa(8(8BZs)x1y1x2y2) rJr rQrnallr"r;rrrr) raw_cellsgridrowsboxesr'grid_rowraw_rowout_rowjrs @r*cells_to_boxesrs(34 ?DEGE  #$s4y=$tw-bEGs8}%A!"S\!17:tDC'[BZ'[$[$[dggdggdggPUVZV]V]P^_`t$ &  W' Lr,c*[[U55$rO)rqr)rs r*df_to_markdownrs KO ,,r,cSSKnSSKJn SSKJn /nUR USUS9n0n[ U5GH up[U SS5n U b [U 5OSn [U S S5n U (a [U 5OSn [U S 05=(d 0n[URS S 5=(d S 5n[U R5n[[U S S5U5nX;aU"[U5U 5unnnUU4X{'X{unnU"UUUU5nUR![#USU SU3UU U UUU[%U5S95 GM U$)zTop-level entrypoint for the spawned process. Imports camelot fresh in the child so OpenCV initialization stays inside the worker and never crosses a fork boundary. rN) page_words)tighten_cell_boxesr)pagesr3r6r_bboxparsing_reportr<r8r?z-p-)r1r3r6r7r<r?rArB)camelotquber.agents.completenessr%quber.core.extractors.camelot.tightenrread_pdfr getattrr5tupler;getrrrrr"r.rq) source_strr3rrrrytables words_cacheidxtableraw_pager6raw_bboxr7reportr<r?rArgpage_hwordss r*camelot_workerrsZ 4H"$C   jf  EF68K' 5&!, ( 4s8}!5'40"*uX 0"5;J4;<EHH%#GE7D$A5I  ")$z*:DA Avu!'K #) 'z5&I   &xr$q6!%)%0  !(8 Jr,c [R"S5n/nSS/n[SUS9nUVs0sH#ofUR[[ U5U5_M% nnUR 5H$uphURURUS95 M& S S S 5 U(d![R"S UR5 U$s snf![a5 [R"SUUURS9 UR5 M[a,n [R"S UURU S 9 S n A MS n A ff=f!,(df  N=f) u Run lattice + stream concurrently in spawn-context subprocesses. A flavor that times out or raises is logged ERROR and contributes zero candidates; the other flavor's output is still returned. If BOTH fail, returns an empty list — the caller decides what to do. spawnrrr) max_workers mp_context)timeoutz7camelot {flavor} timed out after {timeout}s on {source})r3rr#z*camelot {flavor} raised on {source}: {exc})r3r#excNz5camelot produced 0 candidates from BOTH flavors on {})mp get_contextrsubmitrritemsroresultFutureTimeoutErrorr errornamecancel Exception) r# timeout_sctx candidatesflavorspoolr3futuresfuturers r*run_camelot_flavors_parallelrs .. !C)+J&1G s ;tZabZaPV4;;~s6{FKKZab%mmoNF !!&-- -"BC. <*  LfkkZ -c&  M!%!;;     @!!;;   < ;sRD=*CD=.C D=D=;D:D= D:!D5/D=5D::D== E c0UR5(dgUR5HgnUR5RS5RS5Vs/sHo"R5PM nn[SU55(dMg g gs snf)zA Camelot output is content-empty when every non-delimiter cell is whitespace. Lattice's false-positive shells on whitespace-aligned PDFs look like a 1-row grid of empty pipes; this catches them before they reach the classifier or correction stage. T|c3T# UHo=(a [U5SS1::+v M g7f)r N)set)rSr_s r*rU#is_content_empty.. s#=u!-QC:--us&(F)r splitlinessplitany)rBliner_r?s r*is_content_emptyrs| >>  ##%$(JJL$6$6s$;$A$A#$FG$Fq$FG =u= = =& HsB)r#rrr5r$rrlz List[Path])r?r>rlr)rxr5rlr)rrrlr>)rrrr>rlr@)rrrlr)rrr3r2rlList[CamelotCandidate])r#rrr;rlr)rBrrlbool)&rH __future__rmultiprocessingrconcurrent.futuresrrrpathlibrtypingrrr r r r logurur pydanticrrpandasrr2CAMELOT_FLAVOR_TIMEOUT_Sr+r.rqr{rrrrrrrrCr,r*rs(#2AEE% $ % eye>$2A )<4-*^0% %%%P r,