jSrSSKJr SSKrSSKrSSKrSSKrSSKrSSKJ r SSK J r J r J r SSKJr \R "5 \R""\R$SS9 SS KJr SS KJr SS KJrJr SS KJrJr SS KJrJrJ r J!r! SSK"J#r# SSK$J%r% SSK&J'r' \'"5RPRRr*Sr+1Skr,S.Sjr-S/Sjr.S0Sjr/S1Sjr0S2Sjr1S3Sjr2S4Sjr3S5Sjr4S6Sjr5S7S8Sjjr6S9Sjr7S:Sjr8S;Sjr9SS"jrSAS%jr?SBSCS&jjr@SDS'jrASES(jrBSFS)jrCSGS*jrDSHSIS+jjrESBSJS,jjrF\GS-:Xa\F"5 gg)KuIngest quber *fusion* output into the pgvector playground schema. The package's only ingester. It requires the source PDF and two artifacts the quber fusion workflow writes: .unified.json — the unified DoclingDocument (spine: reading-order text + headings, each with page + PDF-point bbox) .tables.json — the corrected ExtractedTable list (LLM-cleaned markdown, title, units, page, normalized regions) The PDF is hashed for the document row and rendered for the footnote lookup agent. The fusion report and three figure-run artifacts are optional inputs, described below. Mapping to the RAG schema, by chunk_type: - text <- unified.json `texts` (bbox normalized to top-left 0..1), and in the grouped stream one record per binding group - picture <- unified.json `pictures` that carry a description or values and are not a logo, icon or watermark - figure_values <- one record per page of figure values no picture claims - table <- tables.json, plus SoM-missed docling tables. A tables.json chunk leads with its section heading and header lines, then the cell-id HTML grid (the markdown when the table has no grid), then its footnote block. Its region is already normalized top-left 0..1. - line_item <- one record per printed line of a gridded table that carries a header depth Grounding is cell-level: each table's `corrected_grid` supplies one GroundedCell per markdown cell — text, a box normalized 0..1 top-left, a provenance status, and the status inspector's note where one was taken. The table chunk's content is an HTML table with an id on every cell (``), so the agent can cite individual cells and the app resolves them to overlays. Every cell is served regardless of status; a cell whose box could not be traced falls back to a box read off its boxed neighbours, then to the table's region, and keeps its status/note so the UI can flag it. A table without a `corrected_grid` gets no cell groundings. Its chunk is its markdown with no cell ids, so it can be cited only as a whole table. Footnotes attach to the chunks that reference them. Each table's and each figure's markers are resolved in tier order: in-crop pairs, then a reading-order scan, then the demand-driven lookup agent. A marker the correction agent judged a cross-reference to a named section resolves to a pointer at that heading rather than to definition text. The resolved text is appended as a footnote block — on the whole-table chunk (all notes, plus unmarked general notes), on each line record whose rendered rows carry the marker, and on each picture chunk. The block is kept out of the embedding. It stays in the stored content, so keyword search matches it. Unresolved markers, unreferenced definitions, and unplaced markers land as tableFlag groundings, quoted verbatim. The fusion report (`.fusion.json`), when given, matches each SoM table to its docling table. That match gives the table chunk its fused caption lines and section heading, places the table in the reading order so its footnote scan starts after it and stops at the next table, and gives a footnote text that directly follows the docling table a parent link to the SoM table's chunk. A table with no match, which includes every scanned table, falls back to its own caption, title, subtitle and units, takes no local section heading, gets no footnote parent link, and scans every line from its page onward for footnotes. The report also names any docling table the Set-of-Mark engine missed (`som_miss`). Those tables exist only in the unified document, so they are ingested from there as plain table chunks — docling's own cell text, table-level grounding — rather than dropped. The figure run (`quber.core.figures.orchestrator`) writes the three optional artifacts. Its scanned tables (`.scanned-tables.json`) are `ExtractedTable`s like any other and join the table engine's own, so a table read off a page image is chunked, grounded and footnoted by the same code. Its run record (`.figures.json`) names the parse tables those replaced, and the parse's reading of a replaced table is then not ingested beside the scan's. Its figure values (`.figure-values.json`) carry the reconciled per-value reading of each figure with per-value provenance. Each value becomes a grounding of its own — ref id `fv-`, its reconciliation status and note, and the value's box on the page (already normalized top-left 0..1; a value with no box of its own borrows its picture's). The values are also listed as `[fv-]`-tagged plain text lines, on the picture chunk they belong to or, for a value with no picture, on its page's figure_values record. The answering model cites a plotted value by that id as precisely as a table cell. A value whose picture is a logo, icon or watermark is listed on no chunk and exists only as its grounding. ) annotationsN)Path)AnyOptionalSequence)loggerINFO)level) FootnoteDef) SCAN_BLOCK)FOOTNOTE_MARKS_FIELDFOOTNOTES_FIELD) FigureValueFigureValueRun)FootnoteResolution absorb_lookupcanonical_markerresolve_footnotes)db)embed_document) get_settingsz Footnotes: >iconlogo watermarkcU(dgURS5(d@[U5R5(d"[R"SUR 5 g[ R"[X5R55$)uAn optional artifact, read if it is there. The figure run writes its scanned-tables file only when it read a table off a page image, and its figure-values file whenever the value step ran, which needs the page cells beside the parse, even if it read no values. A document never put through the figure run has none of its artifacts. A local path that does not exist is not an error — it says the run produced nothing of that kind. An s3:// URI is fetched without that check, so a missing object raises. Ns3://z!no {} to read; the run wrote none) startswithrexistsrinfonamejsonloads_fetch read_textsrcdests >' " "49+;+;+=+= 7C ::fS'113 44c URRSSS9 URS5(a&[R"SSSU[ U5/SSS9 U$UR [U5R55 U$)NT)parentsexist_okrawss3cp)checkcapture_output) parentmkdirr subprocessrunstr write_bytesr read_bytesr%s r(r#r#sqKKdT2 ~~gtT3D :$W[\ K c--/0 Kr*cU(a0U(a)UR5UR5:waUSU3$U=(d U$)uThe section vocabulary a chunk rides with: the page's opening heading joined with the heading in effect at the chunk. A presentation page opens with its subject — 'Commercial Portfolio Geographic Diversification' — and then prints panel labels the parse also calls headings ('U.S.', 'Europe', 'Australia'). Carrying only the latest heading strips the page's subject from every chunk below the first panel label, and a question phrased in the subject's words then never ranks the chunk. Both ride along; when they agree, one.  — lower) page_headingheadings r( _governingr@s;L$6$6$8GMMO$KuWI..  "l"r*cdUS:H=(a% UR5=(a [U5U:H$)u|Is this text the page printing its own number and nothing else? The page's furniture layer is where the parse puts what it takes for page decoration, but its guess is unreliable in the direction that matters: on a financial deck that layer also holds the footnote definitions that qualify the figures, so it cannot be excluded wholesale. A page number is the one thing in it that can be named exactly rather than guessed at — a furniture text whose whole content, read as an integer, is the number of the page it sits on. Anything that fails to match is kept, so a deck whose printed numbering runs off the page index loses nothing. furniture)isdigitint)textpage content_layers r(_is_page_numberrHs( K ' PDLLN Ps4yD?PPr*c URS5=(d 0R[5=(d /Vs/sHEnURS5(dM[URSS5URSS5S9PMG sn$s snf)z?The notes an agent read off the page this figure is printed on.metarEmarkerrKrE)getrr )picns r(_picture_notesrQst776?(b--o>D"D DA 55= H 1552.QUU625FGD  s B.Bc[URS5=(d 0R[5=(d /5$)zThe footnote reference markers an agent read off this figure. Each carries what it points at, so a cross-reference to a named part of the document is never hunted among the lines printed near the figure. rJ)listrNr )rOs r(_picture_marksrTs- #''&/'R,,-ABIr JJr*c U(d [5$URS/5VVVs1sH9nURS/5HnURS5nU(dMUiM! M; snnn$s snnnf)aAThe parse tables a scanned table replaced, from the figure run's record. The fusion report names a table the table engine missed, so the ingest reads that table out of the parse. Where the figure run has since replaced the parse's reading with the scan's, reading it out of the parse again would state one region twice. scanstables table_ref)setrN)r6scantablerefs r(_superseded_refsr]si u GGGR( (DXXh+EIIk*C  + (  s 2A- A-cURS5=(d 0nURS5=(d 0RS5=(d /nU(aUSRSS5$S$)zEThe parse's top predicted class for a picture, or '' if unclassified.rJclassification predictionsr class_namerL)rN)rOrJpredss r(_picture_classrcsT 776? bDhh'(.B 3 3M B IrE-258<< b )::r*c6U(a SUSUSU3$SUSU3$)uOne value line the answering model can cite. The `[fv-n]` tag is the citation contract: the model quotes the id and the app resolves it to that value's own grounding box, so the overlay is the printed value rather than whatever chunk carried the line. Every chunk that serves figure values builds its lines here — the tag must never be reimplemented per chunk kind, because a line without it can only be cited at chunk granularity. [z] z: )fidr values r( _cited_lineris1*.Qse2dV2eW %EQse2eW3EEr*c[U5[;agURS5=(d 0RS5=(d 0RS5=(d SnUR5nU(a/nUHXupVSR SUR UR 455nUR[XWUR55 MZ U(aUS-OSS R U5-nU$) aWhat a picture says, or '' when it says nothing worth retrieving. A chart's plotted values live in the description the page scan wrote, which names the axes, the units, the legend and every plotted figure, so a picture carrying one is a retrievable record like any other. `values` are this picture's reconciled figure values, each with its grounding ref id. They are appended as an id-tagged line per value, so the answering model can cite a plotted value by id, as it cites a grid cell, and the app resolves it to an overlay. Excluded: a watermark, an icon, a logo. They are identity marks repeated on every page with nothing inside to read. A picture with no description and no values is excluded too, having nothing to say. rLrJ descriptionrE c36# UHo(dM Uv M g7fNrf).0ps r( _picture_text..s@':!aAA':  z Values: zValues:  ) rcEXCLUDED_PICTURE_CLASSESrNstripjoinlabelseriesappendrirh)rOvaluesrkrElinesrgvr s r( _picture_textr~s c66GGFO)r..}=CHHPVTVK    D FC88@':@@D LLS8 9)-}$+5AQQ Kr*c 0nUHDup4UR(aMURUR/5RX445 MF /n[ U5Hn/nX&Hcup4SR SUR URUR455nUR[X8UR55 Me SR U5n URUS5n U (a)U R5U R5;aU SU 3n URSU3SUS- SU S45 M U$) u One searchable record per page of figure values no picture claims. An anchored value rides its picture chunk's searchable text, unless the picture is a logo, icon or watermark: that picture has no chunk, and its values are not collected here either, so they exist only as grounding rows. An unanchored value has no chunk to ride and its grounding row alone is invisible to retrieval — a printed number the index never hears of. Each page's unanchored values become one record of id-tagged label-value lines under the page's opening heading, id `figure-values-p`. The id tags are built by `_cited_line`, as `_picture_text` builds a picture's values: the answering model cites the value's own `fv-` id, which resolves to that value's tight box rather than the record's. The record itself carries NO box, deliberately: a citation of the record alone names the page, and the only rectangles ever drawn are the values' own. A synthetic union box was tried and rendered as a misleading figure-sized highlight over the tight one. r;c36# UHo(dM Uv M g7frnrf)roparts r(rq,_unanchored_value_records..<s\1SW[1SrsrtrLfigure-values-p figure_valuesN) picture_ref setdefaultrFrzsortedrw chart_titlerxryrirhrNr=) rpage_first_headingby_pagergr}recordspage1r|r contentsections r(_unanchored_value_recordsrs*9;G}}}   qvvr * 1 13( ; OQGnFC<<\!--!((1S\\D LLS8 9 % ))E"$((3 w}}gmmo= wi0G/%1?EAItU\^bcd! Nr*cJU(dgURS/5HnURS5U:XdMURS5=(d 0/SnURS5(d gURSS5nURUS 5upg[USXg5s $ g) zKThe parse picture's own normalized box, for a value with no box of its own.Npicturesself_refprovrbboxpage_nor?r)rN _norm_bbox)docsizesrrOrrwhs r( _picture_bboxrFs wwz2& 77: + -GGFO+tQ/D88F##HHY*E99UJ/DAd6lA1 1' r*c ^^^^SVs0sHoURU/5_M snm/m[5mS UUUU4SjjmURS05RS5=(d /HnT"URSS55 M T(d/$SHhn[[TU55HJnSUS U3nUT;dM[R "S US 9 TR U5 TRU5 ML Mj T$s snf) awEvery item ref in the document, in reading order. Descends the whole tree. An item's own children are visited straight after it, so a table's footnotes and a picture's labels keep their place in the flow instead of being unreachable. Any text, table or picture the document holds but never links into the tree is appended at the end, so an item is dropped only by a rule that names it, never by an accident of where it sits. Returns an empty list when the artifact carries no body tree. The caller then walks the texts array alone, in its own order, so no picture or group is ingested and no table gets a reading-order position. )textsrWrgroupsc>U(aUT ;agT RU5 URS5RS5upnTRU/5nU[ U5nTRU5 URS5=(d /HnT "URSS55 M g![ [ 4a gf=f)N#//children$refrL)addlstrip partitionrNrD ValueError IndexErrorrz) r\kind_idxitemsitemchildoutpoolsseenvisits r(r"_reading_order_refs..visitiscTk   zz$'11#6  $# S?D 3XXj)/R/E %))FB' (0J'   sB00CCbodyrrrL)rrWrrrzG{ref} is in the document but linked into no parent; ingesting it anywayr\)r\r7returnNone)rNrYrangelenrwarningrzr) rrrir\rrrrs @@@@r(_reading_order_refsrWs2[ [1Z3774$ $1Z [ECUD))&$((4:: eii#$;  /s5;'(AtfAaS/C$]cf 3 )0 JI \sDc80nURS5=(d 0R5Hkup#URS05n[URSS55=(d S[URSS55=(d S4U[U5'Mm U$)z>page_no -> (width, height) from the DoclingDocument pages map.pagessizewidthrrheight)rNrfloatrD)rrkr}rs r( _page_sizesrs*,C!'R..0uuVR TXXgq129c5(TUAV;W;^[^_CF 1 Jr*c URS5URS5pCURS5URS5peUb UbUbUcgURSS5nUS:XaX%- U- X&- U- pOXR- Xb- p[SX1- 5[S[X55[S XA- 5[S [X55S .$) zNormalize a docling prov bbox to top-left origin, 0..1. docling boxes are PDF points with coord_origin BOTTOMLEFT, so t/b are measured up from the page bottom (t > b). Flip Y to a top-left origin. lrtbN coord_origin BOTTOMLEFTgrlefttoprightbottom)rNmaxmin) prov_bboxpage_wpage_hrrtop_pt bottom_ptoriginrrs r(rrs --$immC&8%! c*IMM#,>I |u}):K ]]>< 8F &063E2OVoy'9VC'3C()S%.)c3s+,  r*cfURSS5RSS5RSS5$)N&z&z>replacess r(_escrs- 99S' " * *3 7 ? ?V LLr*c0UbURSS5$S$)zStrip NUL characters: Postgres text fields reject 0x00. They reach us from PDF text layers that encode glyphs like cover-page checkboxes as NUL.NrLrrs r(_pg_textrs%&M199VR ;t;r*cUR[5Sn[R"SSU5n[R"SSU5n[R"SSU5R 5$)zThe text a reader reads: tags dropped, cells separated by ' | '. The footnote block is cut before embedding. It stays in the stored content.rz]*> | z<[^>]+>rlz[ \t]+)splitFOOTNOTE_HEADERresubrv)rrEs r( _embed_viewrsXmmO,Q/G 66#UG 4D 66*c4 (D 66)S$ ' - - //r*cU(a/nURS5=(d /Hn[U[5(aURSS5O [U5nUR S5(dMM[ UR SS5S5nU[U5:a*X&RS5=(d SR5OSnU(dMURU5 M U(aU$URS5URS 5URS 5URS 54Vs/sH o(dM UPM sn$s snf) a9The header lines of a table chunk: the caption nodes fusion attached to the table's docling item, in order (the sentence that introduces the table, its printed name, its units), read from the unified document so the chunk says what the fused document says. A table with no fused item falls back to its own fields.captionsrrL#/texts/rrrEcaptiontitlesubtitleunits) rN isinstancedictr7rrDrsplitrrvrz) rrrr|r\crefrrErps r( _table_headrs  88J'-2-C*4S$*?*?37762&SXD??:..dkk#q)!,-C=@3u:=MEJNN6*0b779SUDt T". Li(!%%.!%% :KQUUSZ^\ b\!`aA\ bb bs 5 E E c^^/n[U5HLupEU(aX$OUmSRUU4Sj[U555nURSUS35 MN SSRU5-S-$)aRender a corrected_grid as an HTML table with an id on every cell. Ids follow `t--` so they are unique per document, short enough for the agent to cite verbatim, and resolve directly back to the grid position they came from. `row_offsets` supplies the true row number of each rendered row when `grid` is a subset (a line-item record rendering header + one line keeps the line's real ids). rLc 3># UH7upSTSTSUS[URS5=(d S5S3 v M9 g7f)z rErLzN)rrN)roccellr table_indexs r(rq_grid_html..sI ) }AaS!BtDHHV4D4J/K.LE R)s?Azzz rtz
) enumeraterwrz)rgrid row_offsetsrowsrrowtdsrs` @r( _grid_htmlrsu DD/)KNqgg $S>   d3%u%& " 4 (< 77r*cUVs/sH5n[U5U:dMXBRS5(dM.XBSPM7 nnXVs/sHofRS5(dMUSPM! nnUVVs/sH(oDHofRS5(dMUSPM! M* nnnU(dU(dgU(a$[SU55[SU55pOU(a USUSpOgU(a$[SU55[SU55pO?U(a7U(a0[S U55US :aUS [S U55pOgXX4$s snfs snfs snnf) uApproximate box for an unboxed cell, read off its boxed neighbors. The column's x-extent (union of boxes in column c) crossed with the row's y-extent (union of boxes in row r) marks the empty spot where the cell sits. A fully unboxed row falls back to the band between the table's top edge and the first boxed row — the header area. Returns None when either axis cannot be derived; the caller falls back to the whole table region. boxNc3*# UH oSv M g7frNrfrors r(rq!_cell_vicinity..s'3aqT3c3*# UH oSv M g7fNrfr s r(rqrs,?3aqT3rrrc3*# UH oSv M g7frNrfr s r(rqrs(4aqT4rc3*# UH oSv M g7fNrfr s r(rqrs-ADqdDrc3*# UH oSv M g7frrfr s r(rqrs 4t!1trrc3*# UH oSv M g7frrfr s r(rqrs3d!dr)rrNrr) rrrregionrcolrrowballbx1x2y1y2s r(_cell_vicinityr"s/%) ODSCHqL=SVZZ=N=36%=DC O$(G ?GDxxKDKGD ?#' K4CCD88E?KDKCK4D K 4 '3'',?3,?)?B F1IB (4((#-AD-A*AB DS 4t 44vay@C3d33B B # P ? Ks'E"E" E" E'& E'6E, E,c U(dgUVVs/sH(o"Ho3RS5(dMUSPM! M* nnnU(a[SU55US::agUSUSUS[SU554$s snnf)a5The table's header area: from the region's top edge down to the top of the first boxed row. Dropped printed text lives there by definition, so it is the honest vicinity for a dropped-text flag. None without a region, when no cell is boxed, or when the first boxed row starts at or above the region's top edge.Nr c3*# UH oSv M g7frrfr s r(rq_header_band..s*TtTrrrrc3*# UH oSv M g7frrfr s r(rqr%s0Dt!1tr)rNr)rrrrrs r( _header_bandr's| #' K4CCD88E?KDKCK4D K 3*T**fQi7 1Ivay&)S0Dt0D-D EE Ls B Bc^0mURS5=(d 0RS/5HmnURS5=(d SR5nU(dM4URSS5nURSS5nUTRU05U'Mo SRU4S j[ T555$) z=Reconstruct a docling table's content as pipe-separated rows.data table_cellsrErLstart_row_offset_idxrstart_col_offset_idxrtc3v>^# UH-mSRUU4Sj[TT555v M/ g7f)rc34># UH nTTUv M g7frnrf)rorrrs r(rq0_docling_table_text...!sDOqQ OsN)rwr)rorrs @r(rq&_docling_table_text..!s+[lUZZDF47ODDDls59)rNrvrrwr)rrrErrrs @r(_docling_table_textr1s&(Dv$"))-< &B--/  HH+Q / HH+Q /$(2q! = 99[fUYl[ [[r*c UR5(dU$SSKnSSKnSSKJn SSKJn U"5nU"[U5SSX"S-S9n/n UHBn UR5n U RU S S 9 U RU R55 MD URURURX55n U cU$[%X R&5$![a!n [ R""S X-S 9 UsSn A $Sn A ff=f) uThe demand-driven last tier: ask the lookup agent to find the still- unresolved markers on the rendered pages following the table. Any failure (no PDF, no credentials, a failed call) leaves the markers unresolved in the exception record — the lookup never takes an ingest down.rN)convert_from_path)PydanticAIFootnoteLookuppngr)dpifmt first_page last_pagePNG)formatzCpage {p}: footnote lookup unavailable; markers stay unresolved: {e})rpe)rasyncioio pdf2imager3quber.agents.footnote_lookupr4r7BytesIOsaverzgetvaluer6lookup unresolved Exceptionrrrfound) resolutionpdf_pathrr>r?r3r4agentimagespngsimagebufreportexcs r(_lookup_unresolvedrR$s ??  /I(*"3x=cuQVjkbklE**,C JJs5J ) KK  'U\\**?*?MN~ \\ 22 \`emsB%C D"C>8D>Dc =^U S:Hn [U[US3- 5n [R"[U[US3- 5R 55n [R"[U[US3- 5R 55n U(a8[R"[U[US3- 5R 55OSn[ U[US3- 5nU R U=(d /5 [[ U[US3- 55n[U 5n[ U[US 3- 5nU(aD[[R"U5R5VVs/sH unnS U3U4PM snnO/n0nUHFunnUR(dMURUR/5RUU45 MH /n/n0n0nU(amUR!S /5HWnUR!S /5nUR!S /5H,nU(dM USUU'URUSSU35 M. MY U R!S/5nU R!S/5n U R!S/5n![#U 5n"U"=(d( [%['U55V#s/sHn#SU#3PM sn#n$Sn%0n&/n'0n(/n)/n*0n+Sn,/n-/n.[)5n/U$GH!n0U0R+S5(GaoU (dM$[-U0R/SS5S5n1U1['U!5:aU!U1O0n2U2R!S5[0:waU2R!S5S:waM/n3U2R!S/5GHn4U4R!S5=(d Sn5U5R+S5(dM6U[-U5R/SS5S5n6U6R!S5=(d SR35n7U7(dMU6R!S5S:XaUU7n%U6R!S5=(d 0/SR!S S5n8U*RU7U845 U+RU8U75 U3RU5U6U745 GM U3(dGMU/R5S!U355 U3SSR!S5=(d 0/Sn9U9R!S S5n:UR!U:S"5un;nV?V@s/sHXun=n6n>U6R!S5=(d 0/Sn?U?R!S#5(dM=[7U?S#U;U<5n@U@(dMVW@PMZ nAn?n>n6n=n@UA(aG[9S$WA55[9S%UA55[;S&UA55[;S'UA55S(.OSnBU3Hun=nCn7U'RU7U:45 M S)R=S*U355nD[?U+R!U:S5U%5mT(a)TRA5WDRA5;aTS+WD3nDURU2R!S,5=(d U0SU:S- WBWDS45 GMU0U/;aGMU0R+S5(aCU%U&U0'['U'5U(U0'U)RU05 U-R['U'55 U0n,GMU0R+S-5(GaU [-U0R/SS5S5nE[CUEUR!UER!S,5=(d S5=(d UR!U0/55nDUD(dGMzWER!S5=(d 0/Sn9U9R!S S5n:UR!U:S"5un;nUb4PM snb-WFVs/sHFnUR!S5=(d SR35(aM2S?UR!S3S54PMH sn-nc[Uc5HAundunenb[dRj"S@U0U:UeUbSA9 URU0SBUd3SCW]WBSDU00UeSUb45 MC GM UR [mUU+55 UHunnURn=(d [qU UUR5nfURrURtURvURSE.ngUR(dSFURx3WgSD'URUSGURxS- Wf(a [{Wf5OSWgUR|UR~UR45 M U R!SH/5Vhs0sHnhUhR!S,5Uh_M ninh[U 5GHun#n6U6R!SIS5n:U6R!SJ5=(d U6R!SK5njSnBUj(aWjunknlnmnnUkUlUmUnS(.nBU6R!SL5=(d /noUo(a [U#Wo5OU6R!SM5n^[U6WiR!UR!U#S55U5npU&R!UR!U#S5S5mU (a,[?U+R!U6R!SIS5S5T5mT(a,[U4SNjWp55(aWpRST5 U6R!SO5=(d /nFU6R!SP5=(d /Vqs/sH8nq[Uq[5(a [SWqSQ9O[R"Wq5PM: nrnqWFVs/sHnUR!S3S5PM sn[OU6R!SR5=(d /5-nRUR!U#S5nsU(R!Us5nQUQb?U)RWs5ntUtS-['U)5:a U(U)WtS-O ['U'5nTU'WQUTnUOU'Vus/sHnuUuSU::dMWuPM nUnuWFVs1sH5nUR!S45S5:XdM[KUR!S3S55iM7 nVn[QWRWrU:WUU*UV5nZUZRZ(a[_WZ[US3- U:5nZWZRTV[s/sHn[U[RLS+U[R`3PM sn[WrVWs/sHCnWUWRL(aMWWR`R35(dM7WWR`PME snW-nvW^(aS)R=WpW^/-5OS)R=Wp5nDWv(aWD[bS)R=Wv5-- nDURSU#3S6U:S- WBWDS45 U6R!SS5nwWo(a-[Ww[,5(d[dRj"STU#U:SU9 GOUWRLSn6n=fs snfs sn?fs snfs snWfs snWfs sn[fs sn[fs snfs snbfs snfs snhfs snqfs snfs snufs snfs sn[fs snWfs sndfs snfs snfs snWfs snfs snnfs sn6fs sn[f!,(df  GN=f)u`chunking` selects the ingestion stream. `flat` (the default) serves every printed line on its own: a stat panel's label lands in one chunk and its number in another, and each text and picture chunk is tagged with only the most recent heading the walk passed — nothing is bundled. A table chunk carries a heading only when the fusion report matches it to a docling table the walk reached, so a scanned table carries none. Technically: every group is a transparent wrapper, single-heading ride-along. `grouped`: a binding group — a key_value_area or a scan block — is served as one chunk so a label travels with its number, and every chunk rides with the page's opening heading joined to the local one. A per-page figure_values record carries the page's opening heading in both streams. The upload app picks the stream per track (`TRACKS` in `quber.playground.app`): the dpt-2 track ingests flat, the dpt-3 track, its default, ingests grouped. groupedz.pdfz .unified.jsonz .tables.jsonz .fusion.jsonNz.scanned-tables.jsonz .figures.jsonz.figure-values.jsonzfv-matchesdocling_table_refs som_indicesrz #/tables/rrrrrLz #/groups/rrr rxkey_value_arearrrEsection_headerrrc3*# UH upo1v M g7frnrf)romref_t_texts r(rqingest..s"GwODe4wrrrc3*# UH oSv M g7f)rNrfr s r(rqr^s95a& 5rc3*# UH oSv M g7f)rNrfr s r(rqr^s7Axrc3*# UH oSv M g7f)rNrfr s r(rqr^s ;U7Urc3*# UH oSv M g7f)rNrfr s r(rqr^s!=u!H+urrrtc3*# UH upo3v M g7frnrf)ro_mrefr\rEs r(rqr^sDGDGrr;rz #/pictures/picturerrGfootnotec3# UH7oRS5(dM[URSS55v M9 g7f)rKrLN)rNr)roms r(rqr^Ps25 ;@aEE(O 1 QUU8R0 1 15s A"ArKrrr[sourcesibling)update)resolvedrF unreferencedz/Attached {} footnote(s) to picture {} (page {})rlfootnote_unresolvedfootnote_unreferencedfootnote_marker_unplacedz'picture {r} (page {p}): {status}: {q!r})rrpstatusqz-fn- tableFlagchunk_id)rxrychartrr figureValuerWrFcontent_region som_regioncorrected_gridmarkdownc3f># UH&nTR5UR5;v M( g7frnr<)rorprs r(rqr^s"JT7==?!'');Ts.1footnote_marks footnotesrM footnote_refs header_rowsutable {i} (page {p}): gridded but carries no header depth — structure correction absent; serving the whole-table record only, no per-line records)rrpc3p# UH,oRS5=(d SR5v M. g7frErLNrNrvrors r(rqr^(s(LHHV,299;;46c3p# UH,oRS5=(d SR5v M. g7frrrs r(rqr^.s(PHHV,299;;r)rrr c3*# UH oSv M g7fr rfr s r(rqr^Ks#8%QaD%rc3*# UH oSv M g7frrfr s r(rqr^Ls"7AQ4rc3*# UH oSv M g7frrfr s r(rqr^Ms$95aqT5rc3*# UH oSv M g7frrfr s r(rqr^Ns%:EqdErrz-line- line_itemrrr tableCell)rrrtrqnote dropped_textz -dropped-header_text_droppedz%table {i} (page {p}): {status}: {q!r})rrprqrrsom_missz?fusion report names {ref} but the unified doc has no such tablerz0SoM-missed table {ref} has no cell text; skippedz #/doclingz3Ingesting SoM-missed docling table {ref} (page {p}))r\rprfilenamez7DELETE FROM ade_playground.documents WHERE doc_key = %szINSERT INTO ade_playground.documents (doc_key, content_hash, filename, page_count, ade_version) VALUES (%s, %s, %s, %s, %s) RETURNING idz quber-fusionT)strictzINSERT INTO ade_playground.chunks (document_id, chunk_id, chunk_type, page, bbox, content, embedding, parent_chunk_id) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)zINSERT INTO ade_playground.groundings (document_id, ref_id, ref_type, page, bbox, position) VALUES (%s, %s, %s, %s, %s, NULL) ON CONFLICT (document_id, ref_id) DO NOTHINGaINSERT INTO ade_playground.groundings (document_id, ref_id, ref_type, page, bbox, position, status, note, cell_text) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s) ON CONFLICT (document_id, ref_id) DO NOTHINGzZIngested {key}: {n} chunks ({t} text, {tb} table, {ln} line item), {cells} cell groundingsc3:# UHoSS:XdM Sv M g7f)rrENrfrors r(rqr^s 3AaDFNaa  c3:# UHoSS:XdM Sv M g7f)rr[Nrfrs r(rqr^s5'QqTW_qq'rc3:# UHoSS:XdM Sv M g7f)rrNrfrs r(rqr^s9'QqT[%8qq'r)keyrPrtblncells)^r#DATA_DIRr!r"r$r)extendr]rrrmodel_validater{rrrzrNrrrrYrrDrr rvrkrrrrwr@r=r~rTrQrHrrKrSrrrlri model_copyrFrmrRrErrrrrrr rrxryrrFrrqrrhrrallinsertrr7r indexanytext_forr"r'rr1rrhashlibsha256r9 hexdigestrconnectexecutefetchonezipdumpsrsuccesssum)doc_key unified_json tables_json source_pdf fusion_jsonscanned_tables_json figures_jsonfigure_values_jsonrchunkingrT pdf_localrrWrPscanned supersededr value_runrPr}rvalues_by_picturergr cell_rowssom_docling_refdocling_to_somrhrefssi all_texts all_pictures all_groupsorderr walk_refsr?table_headings ordered_textstable_positionstable_sequenceheadingsrlast_table_refowner_positions picture_notesgrouped_membersref_idgidxgroupmembersrr[rrEmpagerrrrrdr]rprboxesrr\rrOmarksdefsrrxr3 page_defs page_markers_i_p orphans_seenrposmarkerslaterendtrailing section_keysdown_keyspoolrIrref_typepage0rnotesorphansrquotedflagsrrqr positionrunified_tablesrrr rr!rheadffdefsown_refseqx table_notesdepth label_rowrr line_html line_contentrows_set line_notes seen_keys note_textr line_bboxr cell_bboxband band_bboxfragmentfootnote_flagsr\ missed_refsby_refvectorsrJ page_count content_hashconndoc_idvecr cell_textrs @r(ingestr Cs4)#Gz8 .>#>?I **VL(y 5N*NOYY[ \C ZZ{H',7O,OPZZ\ ]F  6+xWI\2J'JKUUWX  18 I]>^3^_G MM'-R "/,gYVcKd@d"efJ  E 2H'J]?^4^_I %.n.K.KI.V.].]$^_$^DAqCs)Q$^_  CEQ ===  ( ( ; B BC8 L OQGegI')O%'N Ir*A55-r2DeeM2.4*.q'OB'"--d1g2$7GH/+$I77:r*L2&J  $EH%I2GH2GQHQC.2GHIG&(N ,.M&(O "N&(H*,%)N"$OPRM!$O   [ ) )v}}S!,Q/0D(,s:(>Jt$BEyy J.599W3EIY3YG:r2yy(.Bz22c$++c1"5a"89:f +224 55>%55"GUU6]2rdA6::9aHEOOT5M2&11%>a/#3$  " ""Gw"G GAJqM%%f-5"q9DHHY*E99UJ/DAq(/'.OE1e55=0RD!4A55=%QvY15A '.   9599777 ;U ;;!!=u!==  $+r4$$dE]3$+iiDGDDG !3!7!7r!BGLG7==?'--/A$IU7)4 NNEIIj1;VVUQYPTV]_cd e  _ $    [ ) )%,N6 "&)-&8OF #  ! !& )  " "3}#5 6#N    ] + +s6==a#8#;<=C#&**377:+>+D"EjIZI^I^_egiIjGGGFO+tQ/DHHY*E99UJ/DAq=AXXf=M=M:dhhvr2Aq9SWDPWj!3!7!7r!BGL]dG7==?'--/A$IU7)4"3'E!#&D&&s='9:$$c'lE4MHZ%[\ NNCGGJ/969eaiQUW^`de f   ,, &--Q'*+ #i.  cNf #**,gf %"q)A& $ $Gu ."--eT:!N u )  4o(>(D" E E dE]+=Bj=P##N$8b9VZ J !Nyy +19=&9I9Iz$((62.15tLS*/33E2>HY`-4PTPZPZP\9\WIU4&)bfj)3VVUQYgW]^_CZ/1I(*L&3"E4UB'..t4su-445 ;@5  '4 *-L*7&udE305I1x&1552&It +7Oqq3wO7eAhS%7 S)GLku!PQPUPUV\P]ajPj=(x)<=u k9==1$QXX.=Dz }}UB/ /!3CAHH3MU]3]A/  'weXxQ]^ ( $,, -A88w&*:188*D*P\\(I)>\?@-  J112j556    +JgYdCS8SUZ[J6=en3%tV5?5H5HI5HAHH:U166(+5HI &5)99 GEN KKI3u:W]_d e ((A"188,Cl&&uce44 8T   eS\ * NNahhZq1779 : )2<1F1F G1FA#Q '1F G?FGwV'0wG HAg,"335B+QUU8R-@A  $-U#3 A NN9VuU[_e    84s#[% F?SU[]acij  $4_+8n NN,]T>ZXZ9[["%%a,!!'* ? &&w/C7S00 sQw 78'  %S-H#0B=aAaDEM=HBGLku!PQPUPUV\P]ajPj=(x)<=u k&wuhR^_  +JgYdCS8SUZ[J?C#"2 MM#& * 3 3AEE(B4G HI y"))QUU8_,=U9+*NO Odii 6K$KKL14HeH !$#8%#8 8""7"77!$$95$9!9"%%:E%:": 6!%{EAIy,ZcdecfXgh[-l oFAs$S>4hhuoKaF)KZ]SVCFSVsSTvVcg  A3as!A3# ! !!9QCI*((.B   *&.D&)^bT!WT!WtAwRVWXRYZhl $QUU>%:%@bAKAx   s)A3'AI9QC1)  B 2<1F1F G1FA#Q '1F GR\RiRijRiQ'AHH:Qqvvh)?)E)E)GHRij kJOh%QSTSXSXY^S_@*AEE(B,?@%h i  $-^#< A NNBa5Y_ci j   s$qcNAI9QC1  $=}"^ZZ 2. .uuV} * uu126 7 .  14"0EF0E1!%% #Q&0EF+ 23C 3Ay`fij)!,GQWZ[EE&M)bT1-DHHY*E$((b1G$%7%;%;E2%FP7==?'--/A$IRy199UJ/DAq=AXXf=M=M:dhhvr2Aq9SWD NNiu-w 4RVW X KKMSVZ_ `)44PWnW'&J'Q{1Q4'8'&JK\^G 778R DC488J/CgYd3CHU#tJ>>)"6"6"89CCEL  NQXPZ[ll ;lHj. I  (* QEHRYbfEg A 5$**T2BHWDUWZ\bc   LLC5$**T2BC  Fh"QZ LFHeT8VT9 LLC JJt$JJx(TNY'   QZ9 ^ NNd  g, 3 3 3 5'5 5 9'9 9)n w<M `NIX^J8l> J< HGXN8 7$CkPS \+B,Iz Hjh( G6'K sN1Aw5Aw$AyZAy Z- Ay bAyb&5Ayc#Ayc=AyfAy f0 Ay g'Ay&nAy+ qD(Ay0y0 Ay?c ,SSKnURSS9nURSSSS9 URSSS S9 URS SS S9 URS SS S9 URSSSS9 URSSSS9 URSSSS9 URSSSS9 URSSSS9 URSSS/SSS9 URSSS S!9 URU5nUR(a+[ R "5 [R"S"5 [URURURURURURUR UR"UR$UR&S#9 g)$Nrz?Ingest quber fusion output into the playground pgvector schema.)rkz --doc-keyTz#opaque storage key for the document)requiredhelpz--unified-jsonz.local path or s3:// URI to .unified.jsonz --tables-jsonz-local path or s3:// URI to .tables.jsonz--pdfz)local path or s3:// URI to the source PDFz --fusion-jsonztlocal path or s3:// URI to .fusion.json; enables ingesting SoM-missed docling tables from the unified document)defaultrz--scanned-tables-jsonzlocal path or s3:// URI to .scanned-tables.json; the tables the figure run read off a page image, ingested alongside the table engine'sz--figures-jsonzlocal path or s3:// URI to .figures.json; names the parse tables a scanned table replaced, so the parse's reading of them is not ingested tooz--figure-values-jsonzlocal path or s3:// URI to .figure-values.json; the reconciled figure values, grounded per value and listed on their picture chunksz --filenamez,original PDF filename, kept for presentationz --chunkingflatrTuingestion stream: flat serves every printed line as its own chunk — the dpt-2 stream's byte-stable behavior (default); grouped is the dpt-3 track's — binding groups served as one chunk, page heading joined into the ride-along)choicesrrz --init-schema store_truez#(re)create the schema before ingest)actionrzApplied schema)rrrrrr)argparseArgumentParser add_argument parse_args init_schemar apply_schemarrr rrrpdfrrrrrr)argvrrpargss r(mainrs,mnANN;4YNZNN#d9iNjNN?T8gNhNN7T0[N\NN> NNR NNU NNO NN<4bNcNN#% NN?<>cNd << D  '(     $$ 44&&22 r*__main__)r& Optional[str]r'rrOptional[dict])r&r7r'rrr)r>r7r?r7rr7)rEr7rFrDrGr7rbool)rOrrzlist[FootnoteDef])rOrr list[dict])r6r!rzset[str])rOrrr7)rgr7r r7rhr7rr7)rf)rOrr{!Sequence[tuple[str, FigureValue]]rr7)rr$rzdict[int, str]rz>list[tuple[str, str, int, Optional[dict], str, Optional[str]]])rrrdict[int, tuple[float, float]]rr rOptional[dict[str, float]])rrr list[str])rrrr%)rzdict[str, Any]rrrrrr&)rr7rr7)rr rr )rr7rr7)rrrr!rr#rr'rn)rrDrlist[list[dict]]rzOptional[list[int]]rr7) rr(rrDrrDrOptional[tuple]r+Optional[tuple[float, float, float, float]])rr(rr)rr*)rrrr7)rIrrJrrrDrr)NNNNNr)rr7rr7rr7rr7rr rr rr rr rr rr7rrD)rzOptional[list[str]]rr)H__doc__ __future__rrr!rr5syspathlibrtypingrrrlogururremoverstderrquber.agents.llm_clientr quber.core.figures.dpt3.blocksr quber.core.figures.graftr rquber.core.figures.modelsrrquber.core.fusion.footnotesrrrrquber.playgroundrquber.playground.embeddingrquber.settingsr playgrounddata_dirrrrur)r#r@rHrQrTr]rcrir~rrrrrrrrrrr"r'r1rRr r__name__rfr*r(r>s6Sj#  **  3::V$/5JA  5' > $ $ - - #95& # QK&; F:&4&&&D&R 4CP"2j.M< 0c*8( "'*4C0> F  F$3 F0 F \3H"&)-"&(,"w  w w w  w   w  ' w  w &w w w  w t;| zFr*