=iGdZddlZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl mZmZddlmZdd lmZdd lmZmZdd lmZdd lmZmZGd deZGddeZGddeZGddZy)z Table Inference with LLM Uses PydanticAI with various model providers to analyze tables and generate meaningful titles and descriptions based on document context. N)datetime)AnyDictListOptional)logger) BaseModelField) AgentFactory) ModelProvider)EnhancedTableParser TableMetadata) load_prompt)Config setup_loggingceZdZUdZedZeed<edZeed<edZ e eed<ed d Z e eed <ed Z eed<ed d Ze eed<ed d Ze eed<edZeed<y ) TableContextz!Input context for table analysis.z#3-4 sentence document-level summary descriptionexecutive_summaryz#Title generated by procedural rulesprocedural_titlezIndividual headers foundheadersNzText that describes the table)defaultrdescriptive_textzFirst few rows of the table table_previewzText before the tablepreceding_textz-Extended page-level context including headers page_contextzPage where table appears page_number)__name__ __module__ __qualname____doc__r rstr__annotations__rrrrrrrrrint>/home/mande/repo/quber/src/quber/processors/table_inference.pyrrs+"/TUsU!.STcT+EFGT#YF&+DFe&fhsmf+HIM3I$)$D[$\NHSM\"'Bq"rL(3-r)CDKDr(rcJeZdZUdZedZeed<edZeed<y) TableInfoz'LLM-inferred information about a table.z"Clear, concise title for the tablertitlezJFunctional description of what the table contains and how it might be usedrN) r r!r"r#r r,r$r%rr'r(r)r+r+&s)1#GHE3H`Kr(r+c.eZdZUdZedZeed<y)ExecutiveSummaryz Executive summary of a document.z.3-4 sentence executive summary of the documentrsummaryN)r r!r"r#r r/r$r%r'r(r)r.r./s*%UVGSVr(r.cbeZdZdZ ddeedeedeefdZ d de deed eed e ee ffd Z dd e e d ee eefd eedeed e e ee ff dZded efdZd!deeded efdZd"deded efdZd efdZded e ee ffdZde ee fdefdZde ee fdefdZy)#TableInferenceProcessorz8Process tables with LLM inference for enhanced metadata.Nprovidermodelconfigc|xstj|_||_||_t d|jj |jjtdd}t|jj |_ |jjt||||_ tjd|xsdd |xsd y ) a? Initialize the processor with specified configuration. Args: provider: Model provider to use (auto-detected if not specified) model: Specific model to use (uses default if not specified) config: Configuration object (loads from environment if not provided) ztable-inference) service_nameenable_logfire log_leveltable_inferencesystem)r7 output_type system_promptr2r3z3Initialized TableInferenceProcessor with provider: autoz , model: rN)rloadr4r2r3rr7r8rr agent_factory create_agentr+agentrdebug)selfr2r3r4r=s r)__init__z TableInferenceProcessor.__init__8s-      *;;55kk++ $$5x@ *9S9ST''44!' 5    A(BTfAUU^_d_qhq^r s r(metadatarrreturnc zKt|xsd|jxsd|jxsg|j|j|j ||j } |j|}|jjretjd|j|j 5|jj|d{}|j}dddn/|jj|d{}|j}t!j"d|j|j|j |j|j|j|j j*|j,|j|j.|j0|j2dd S7#1swYxYw7#t$$rZ}t!j&d|jd |t)|jxsd |jd zd }Yd}~d}~wwxYww)aY Analyze a single table with LLM inference. Args: metadata: Table metadata from enhanced parser page_context: Extended context from the page executive_summary: Document-level executive summary Returns: Dictionary with both procedural and LLM-inferred information zNo document summary availablezUntitled Table)rrrrrrrr analyze_table)table_idpageNzSuccessfully analyzed table zLLM inference failed for table z: zTable zUnable to generate description)r,r)rowscols first_cell) rJrKrrrr llm_titlellm_descriptiontable_markdownrF)rconsolidated_headerrrrRrr_format_promptr4r7logfirespan table_indexrBrunoutputrrC Exceptionerrorr+r,rrMrNfirst_cell_content) rDrFrrcontextpromptresultllm_infoes r)rIz%TableInferenceProcessor.analyze_tablecs/R3R%99M=M$$*%66"11#22% ,,   ((1F{{))\\/H> 9F%}}Hml $zz~~f55!== LL78L8L7MN O!,,(( ( < <'' ( 9 9&55!'33&55  &99  !:ml6   LL:8;O;O:PPRSTRUV W 22Yx?S?SVW?W>X6Y<H siAH;AG2GGG"'G G 2G.analyze_with_limitsR y,001E1EF !//,HYZZ!yyZ!yyysVA1AA12AAA A1AA1AA1A."A% #A.*A1analyze_tables_batch) total_tablesrdN) r4max_concurrent_tablesasyncio SemaphorerrUrVlengather) rDrbrcrrdrirFtasksresultsrhs ` `` @r)rjz,TableInferenceProcessor.analyze_tables_batchs&(L4;;+L+L%+ %%n5  [} [ [ ?LLm(#H-mL \\ "]1CTb $NNE22G M 3 s<A B:B&#B:B-B+B-B:+B--B72B:r]cd|jd|jd|jxsdd|jd|jrdj |jndd |j xsdd |jd S) z-Format the context as a prompt for the agent.z Analyze this table from page z&: === DOCUMENT EXECUTIVE SUMMARY === z === EXTENDED PAGE CONTEXT === z"No extended page context availablez4 === IMMEDIATE TABLE CONTEXT === Procedural Title: z Headers Found: , Nonez= Preceding Text (chronological order, furthest to closest): z Note: The LAST element in preceding text may be a footnote from the previous table. [Table] markers indicate preceding tables. === TABLE DATA (may be truncated at page boundary) === z Based on ALL context levels (document summary, page context, procedural headers, and table data), extract the most appropriate title and provide a functional description for what this COMPLETE table represents. )rrrrrjoinrr)rDr]s r)rTz&TableInferenceProcessor._format_prompts%1123  ==>?++,-.5oo '//*6JK !6"#  ) r(markdownmax_rowsc|sy|jjd}|dt|dzt|}dj |S)z$Get preview of table (first N rows).zNo table content available Nr )stripsplitminrorv)rDrwrxlines preview_liness r)_get_table_previewz*TableInferenceProcessor._get_table_previewsJ/ &&t,=HqL#e* => yy''r(r char_limitc|dz }g}d}g}|jD]}t|ds|j|k(s t|ds-|js:|jj } | sWt|dr"|j d|j d| n|j | |t| z }||k\sn||dzkr|dkDrg} |dz } |jD]g}t|ds|j| k(s t|ds-|js:|jj } | sW| j | i| r>d j| d d } t| d kDrd | dd z} |j d| z|r&|j dd j|d dz|rdj|SdS)a Extract extended context from the page containing the table. Args: document: The Docling document object page_number: The page number (1-indexed) where the table appears char_limit: Maximum characters to extract for context Returns: String containing page-level context including headers and surrounding text rLr_pagetextlevelz [Header Lz] gffffff?rzNi...i z=== Previous Page Context === z=== Current Page Content === z zNo page context available) iterate_itemshasattrrrr{appendrrorv) rDdocumentrrpage_idx context_parts char_countcurrent_page_textelementrprev_page_text prev_page_idx prev_texts r)_extract_page_contextz-TableInferenceProcessor._extract_page_contexts?  --/Gw(W]]h-F7F+ "<<--/D"7G4-44yrRVQW5XY-44T:"c$i/ %3!0  S( (X\N$qLM$1137G,-1Ow/GLL&||113*11$7 4 IInRS&9: y>C' % $%(8 8I$$%F%RS   0499=NsPR=S3TT .;v{{=)[@[[r(cHK|j}|jjtd|j|j }d|d} |j jrNtjd5|j|d{}|jjcdddS|j|d{}|jjS7R#1swYyxYw7(#t$r"}tjd|Yd}~yd}~wwxYww) z Generate a 3-4 sentence executive summary of the entire document. Args: document: The Docling document object Returns: Executive summary string zCYou are analyzing business documents to create executive summaries.r;a5 Analyze this complete document and provide a 3-4 sentence executive summary. The summary should describe: 1. What this document is (type of document, purpose) 2. Who it is about (company, organization, or subject) 3. What are the major points being made (key metrics, themes, or findings) === DOCUMENT === zg Provide a concise executive summary that captures the essential nature and content of this document. generate_executive_summaryNz&Failed to generate executive summary: z#Unable to generate document summary)export_to_markdownr@rAr.r2r3r4r7rUrVrXrYr/rZrr[)rDrdocument_markdown summary_agentr^r_ras r)_generate_executive_summaryz3TableInferenceProcessor._generate_executive_summary-s %779**77(_]]** 8    9{{))\\">?#0#4#4V#<'-'<'M cs?AJ$J2BJ$8J9CJ$ J" CJ$J J$"J$data output_pathct|dd5}tj||dddddtjd|y#1swY"xYw) zSave results as JSON.wutf-8encodingr F)indent ensure_asciiNzJSON results saved to: )openjsondumprrC)rDrrfs r) save_jsonz!TableInferenceProcessor.save_jsonsD +sW 5 IIdAae <6 .{m<=6 5s A  Ac lg}|jd|jd|jd|d|jd|jd|d|jd|jd|jdd d |jd d |jd|jd |d |jd|dD]}|jd|ddzd|dd|jd|jd|dxsd|jdr&|jddj|d|jdr|jd|dddd |jd!|jd"|d#|jd$|d%|jd&|jd'|d(d)d*|d(d+d,|d(jd-r|jd.|d(d-dd/d |jd0rq|jd1|d0jd2dd3}|D]}|j|t |d0jd2d3kDr|jd4|jd5t |d6d785}|j d2j|dddtjd9|y#1swY"xYw):zSave human-readable summary.z# Table Analysis Reportz**Document:** rz **Date:** rz **Model:** runknown/r3z**Total Tables:** rkrz ## Table rJrLz (Page rK)z ### Procedural Extraction:z- **Consolidated Headers:** rrurz- **Individual Headers:** rtrz- **Descriptive Text:** Nrz ### LLM Analysis:z - **Title:** rPz- **Description:** rQz ### Metadata:z - **Size:** rFrMu rows × rNz columnsrOz- **First Cell:** 2rRz ### Table Preview:rzz)| ... | ... | (additional rows omitted) |z ---rrrzSummary saved to: ) rrgrvr|rorwriterrC)rDrrr~tablerliners r) save_summaryz$TableInferenceProcessor.save_summarys4 ./ R ~d:&6%789 R z$'8"9!:;< R {488,)?@A R(^E LL;uZ'81'<&=WU6]OSTU V LL7 8 LL7>P8Q8[U[7\] ^yy# 9$))E)DT:U9VWXyy+, 7>P8QRVSV8W7XX[\] LL. / LL={);(<= > LL.u5F/G.HI J LL* + LLuZ0895CTU[C\B]]ef Z $$\2 1% 2CL2QRUSU2V1WWZ[\yy)* 34 %&6 7 = =d CBQ G )DLL&*u-.44T:;a?LL!LM LL !C$F+sW 5 GGDIIe$ %6 )+786 5s (!L**L3)NNN)NN) )i)r r!r"r#rr r$rrErrrrIrr&rjrrTrrrrrrr'r(r)r1r15sB-1##' ) =)) })  ) Xos? %? 5=c]? ^fgj^k? c3h? H37+/(, &M*& S#X/&$C= & ! & d38n  &Pls8(8C=(C(QT(:\3:\C:\[^:\x/9S/9b6 s6 tCH~6 p>d38n>3> 39c3h39c39r(r1) r#rmrrrtypingrrrrrUlogururpydanticr r agentsr agents.factoryr corerrpromptsrutilsrrrr+r.r1r'r(r)rsj ,,%!*5!) E9 E WyW Z9Z9r(