j6 S r SSKJr SSKJrJrJrJrJr SSK J
r
J r \\ \
4 r " S S\
5 r " S S\
5 r " S S
\
5 r\S r " S S
\
5 r " S S\
5 r " S S\
5 r " S S\
5 r " S S\
5 rg)ah What the figure workflow produces: one record per scanned page, and one per figure.
A `FigureRun` is the run record, written as `.figures.json`. It holds one
`PageScan` for every page the run scanned, including a page that came back
with nothing. A page whose scan or reading failed has no `PageScan` and is
reported in `errors` instead. Either way, a nominated page never disappears
between nomination and output. A chart in a financial document is often where
a number appears that appears nowhere else, so a page that was nominated and
produced nothing has to be visible.
A `FigureRecord` is one figure the scan returned, with the box the scan drew
around it. On the dpt-2 path its text is the scan's reading exactly as
returned. On the dpt-3 path it is the figure's description followed by its
value tables as markdown rows (`dpt3.digest.figure_text`). The text is not
parsed into fields: a chart description has no fixed shape, and two bar charts
in one response used different field names in a different order, so a parser
written against what has been seen encodes one shape and misreads the next.
A `ScannedTable` is one table the scan returned. Unlike a chart, a table is
kept as structure: a grid, with a box for each cell where the scan gave one.
Only the grids the capture step takes go on to the correction and grounding
every other table goes through (`capture.capture_tables`).
`ChartContext` is the rest of the page's returned text: on the dpt-2 path every
text and marginalia chunk, on the dpt-3 path every node with text that is
neither a figure nor a table. That is where the chart titles and the notes at
the foot of the page are, alongside the page's body text. It is kept as
returned and attached to no chart. One note commonly serves several charts on
a page, and matching a superscript to the note it points at is a separate
step, the same way it is for tables.
) annotations)DictListLiteralOptionalTuple) BaseModelFieldc \ rS rSr% Sr\" SS9rS\S' \" SS9rS\S ' \" S
S S 9r S
\S' \" S
SS 9r
S\S' \" S
SS 9r S\S' \" S
SS 9r S\S' Sr
g
) FigureRecord/ z One chart a page scan returned.z+1-based source page the chart is printed on descriptionintpagez;The scan's reading of the chart, stored exactly as returnedstrtextNz+Normalized box around the chart on the pagedefaultr
Optional[Box]boxzCThe scan's own id for this figure, for tracing back to the response
Optional[str]chunk_id@The scan job that produced this record (see the page's PageScan)job_idzThe picture in the refined parse this record was grafted onto. Absent when the figure overlapped no picture, which the run reports as an error. picture_ref )__name__
__module__ __qualname____firstlineno____doc__r
r __annotations__r r r r r __static_attributes__r 7/home/mande/repo/quber/src/quber/core/figures/models.pyr r / s )"OPD#P"_`D#` t9fgC g #"g Hm ""d FM "'
O"K r% r c \ rS rSr% Sr\" SS9rS\S' \" SS9rS\S ' \" S
S9r S \S ' \" S
SS9r
S\S' \" S
SS9r S\S' \" S
SS9r S\S' \" S
SS9r
S\S' \" S
SS9rS\S' \" S
SS9rS\S ' S rg
) ScannedTableD u4 One table a page scan returned, as a grid with a box on every cell.
A table comes back shaped like a table: an HTML grid with an id on every
cell, and a box for each of those ids. That is a different shape from a
chart, which comes back as a description with no internal locations at all,
and it is the right shape for the element it belongs to.
The grid is dense — one entry per row and column, blank where a spanning
cell covers a position — so it drops straight into the same correction and
grounding steps every other table in the document goes through.
z+1-based source page the table is printed onr r r z$The returned grid, rows of cell textzList[List[str]]cellszTNormalized box per cell, shaped exactly like `cells`; absent where none was returnedzList[List[Optional[Box]]]
cell_boxesNz%Normalized box around the whole tabler r r z The scan's own id for this tabler r r r zThe table in the refined parse this grid replaced. Absent when the returned table overlapped no table in the parse, which the run reports as an error. table_refzThe picture in the parse this grid was printed over, when the parse detected the region but filed it as a picture rather than a table. The grid becomes a new table beside that picture.r z6The extracted table this grid produced, by its addresstable_idr )r r r r! r" r
r r# r* r+ r r r r, r r- r$ r r% r& r( r( D s
"OPD#P "/U VE?V,1j-J) t9`aC a #D>` aHma !"d FM %
Y I} "'
-"K $"Z Hm r% r( c \ rS rSr% Sr\" SS9rS\S' \" SS9rS\S ' \" S
S9r S\S ' \" S S
S9r
S\S' \" S SS9r S\S' Sr g ) ChartContexto zKA title or note the scanned page returned, kept as returned and unattached.z*1-based source page the text is printed onr r r z@The chunk type the scan assigned, as returned (text, marginalia)r kindz$The text, stored exactly as returnedr Nz*Normalized box around the text on the pager r r z The scan's own id for this chunkr r r )
r r r r! r" r
r r# r1 r r r r$ r r% r& r/ r/ o sY U"NOD#O"deD#e"HID#I t9efC f #D>` aHmar% r/ )figurestablesemptydroppedc T \ rS rSr% Sr\" SS9rS\S' S\S' \" S S
S 9rS \S
' \" \ SS9r
S\S' \" \ SS9r S\S' \" SSS 9r S\S' \" SSS 9r
S\S' \" SSS 9rS\S' \" SS S 9rS \S ' \" S S S 9rS!\S"' \" SS#S 9rS\S$' \" \ S%9rS&\S'' \" \ S%9rS(\S)' \" \ S%9rS*\S+' S,rg)-PageScan| a One nominated page and what became of it.
`status` says which: `figures` for a scanned page that returned at least one
figure, `tables` for one that returned no figure but at least one table,
`empty` for a scanned page that returned neither, `dropped` for a page the
filter answered no on, which is never scanned and so is never billed.
`figures` does not promise a chart. Nothing here filters on what a figure
contains, and a page's logo comes back as a figure of its own, so a page can
reach `figures` on the strength of its letterhead alone.
`reused` marks a page whose records come from a scan a previous run already
paid for. `credits` is what that scan cost when it was made, so the two
together separate what this run spent from what it read.
1-based source pager r r
PageStatusstatus z0Why a page was dropped; empty for a scanned pager r reasonz,The picture classes that nominated this pagedefault_factoryr List[str]picture_classeszETables in the parse that nominated this page, read off the page image
table_refsNz)The scan job id, absent on a dropped pager r z The model requested for the scanmodelz#The model version the scan reportedversionz,Credits the scan was billed when it was madezOptional[float]creditsFz>True when the records come from a scan an earlier run paid forboolreusedz:Filename of the raw response this page's records came fromresponse_artifactr? List[FigureRecord]r2 List[ScannedTable]r3 zList[ChartContext]contextr )r r r r! r" r
r r# r= listrA rB r rC rD rE rG rH r2 r3 rL r$ r r% r& r7 r7 | s "78D#8 0bcFCc!& *X"OY " [ J "$ r@ fragment_idsz1Normalized box of the fragment printing the valuer r r )r r r r! r" r
r r# r rT rU rV rW r; rX rM rY r r$ r r% r& rR rR s "78D#8!&"Y"K R5Z[K[#PQE3Q0_`FC`#\]E3]$JKFCK :]^D-^ # *Y L) t9lmC mr% rR c \ rS rSr% Sr\" SS9rS\S' \" \S9r S\S ' \" \S9r
S
\S ' \ SS j5 r \ SS
j5 r
Srg)FigureValueRun zLEvery reconciled figure value one run produced, written as its own artifact. The source document's base namer r documentrI zList[FigureValue]valuesr@ errorsc : [ S U R 5 5 $ )Nc 3 H # U H oR S :X d M Sv M g7f
reconciled Nr; .0vs r& ,FigureValueRun.reconciled.. FkXX -E11k " "sumr_ selfs r& rd FigureValueRun.reconciled FdkkFFFr% c : [ S U R 5 5 $ )Nc 3 H # U H oR S :w d M Sv M g7frc rf rg s r& rj )FigureValueRun.flagged.. rl rm rn rp s r& flaggedFigureValueRun.flagged rs r% r Nreturnr )r r r r! r" r
r^ r# rM r_ r` propertyrd rw r$ r r% r& r[ r[ s^ V&GHHcH %d ;F ; d3FI3
G G G Gr% r[ c \ rS rSr% Sr\" SS9rS\S' \" \S9r S\S ' \" \S
S 9r
S \S
' \" \SS 9r S\S' \ SS j5 r
\ SS j5 r\ SS j5 r\ SS j5 r\ SS j5 r\ S S j5 r\ S S j5 r\ S S j5 rSrg) FigureRun zMEverything one figure run produced, for the CLI to print and callers to read.r] r r r^ rI zList[PageScan]scanszYSurfaced problems: a chart that matched no picture in the parse, a page whose scan failedr> r@ r` zdText taken out of the parse as a read figure's furniture, so a completeness check can account for itzList[RemovedText]removedc , [ U R 5 $ )zPages with a scan record, the same count as `scanned`.
It is not the count of pages the parse nominated. It includes pages
added by `pages="all"` and stored dpt-3 pages, and leaves out a
nominated page whose scan failed.
)lenr rp s r& nominatedFigureRun.nominated s 4:: r% c : [ S U R 5 5 $ )zKPages with a scan record, whether this run paid for the scan or reused one.c 3 H # U H oR S :w d M Sv M g7fr5 re Nrf rh ss r& rj $FigureRun.scanned.. BjHH ,A11jrm ro r rp s r& scannedFigureRun.scanned BdjjBBBr% c : [ S U R 5 5 $ )zPages with a scan record that this run submitted.
A page that was billed but whose dpt-3 response could not be read has
no record, so it is not counted.
c 3 n # U H+ oR S :w d M UR ( a M' Sv M- g7fr )r; rG r s r& rj &FigureRun.submitted..
s# SjHH ,A1!((11js 55 5r rp s r& submittedFigureRun.submitted s SdjjSSSr% c : [ S U R 5 5 $ )zYPages whose status is `figures`, after the graft dropped the records over page furniture.c 3 H # U H oR S :X d M Sv M g7f)r2 re Nrf r s r& rj )FigureRun.with_figures.. r rm r rp s r& with_figuresFigureRun.with_figures r r% c : [ S U R 5 5 $ )z/Scanned pages that returned at least one table.c 3 J # U H oR ( d M S v M g7f)re N)r3 r s r& rj (FigureRun.with_tables.. s 5jHH11js # #r rp s r& with_tablesFigureRun.with_tables s 5djj555r% c p U R VVs/ s H oR H o"PM M snn$ s snnf )z4Every figure record the run produced, in page order.)r r2 )rq r cs r& r2 FigureRun.figures s* ::9:ayy!y:999 2c p U R VVs/ s H oR H o"PM M snn$ s snnf )z9Every table the run read off a page image, in page order.)r r3 )rq r ts r& r3 FigureRun.tables s* ::8:axx!x:888r c : [ S U R 5 5 $ )zjCredits billed for the pages counted by `submitted`.
A reused scan was paid for by the run that made it.
c 3 p # U H, oR ( a M UR =( d S v M. g7f)g N)rG rE r s r& rj $FigureRun.credits..) s# HZxx#199##Zs 6 6r rp s r& rE FigureRun.credits# s HTZZHHHr% r Nry )rz rJ )rz rK )rz float)r r r r! r" r
r^ r# rM r r` r r{ r r r r r r2 r3 rE r$ r r% r& r} r} s W&GHHcH !$ 7E>7 oFI "' z"G
C C T T C C 6 6 : : 9 9 I Ir% r} N)r"
__future__r typingr r r r r pydanticr r
r r Boxr r( r/ r: r7 rO rR r[ r} r r% r& r s @ # 7 7 % 3:9 *(9 (Vb9 b <
=
'>y '>Ta) a.n) n8
GY
G @I @Ir%