j d %SrSSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK J r SSK J r JrJrJrJrJr SSKJrJrJr SSKJrJrJrJr SSKJr SS KJr SS K J!r! SS K"J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r* SS K+J,r, SS K-J.r/ SSK-J0r1 SSK2J3r3J4r4J5r5 SSK6J7r7J8r8 SSK9J:r: SSK;JjrhSS?jriSS@jrj\FRS<5SSAj5rl\FRSB5SSCj5rmSSDjrnSSSEjjroSSFjrp"SGSH\5rq\FRSI\\qSJ9SSKj5rr0rsSL\`SM'\R"5ruSNrv\R"\<"5RR5rySO/SPQ0rzSQSRSS.STSUSS.SV.r{SQr|SWSX1r}SSYjr~"SZS[\5rSS\jrSS]jrSS^jrSS_jrSS`jrSSajrSSbjr\FGRSc\SJ9SSSSSSSS\|4 SSdjj5r\FRSe\SJ9SSSfjj5rSSgjrSShjrSSijr\FGRSj\SSJ9SSkj5r\FGRSl5SSmj5r\(GR"5r\3"\Sn9r"SoSp\5r\FGRSq\4SJ9SSrj5rSSsjr\FRSt\4SJ9SSuj5r\FRSv5SSwj5rSSxjr\FRSy5SSzj5r\FRS{5SS|j5rg)u FastAPI backend for the answering playground. Serves the single-page UI and these endpoints: GET /api/documents -> list ingested documents: filing metadata, derived label, overlap and review flags GET /api/document/{id}/pdf -> the source PDF (for PDF.js) GET /api/document/{id}/flags -> the document's review queue, in page order PATCH /api/documents/{id} -> edit metadata (title, folder, filing fields); organization only, never identity DELETE /api/documents/{id} -> remove the document outright (hard delete; 409 while its ingest or a batch run against it is still going) POST /api/chat -> retrieve -> Opus answer -> grounded refs POST /api/chat/stream -> the same answer as server-sent events POST /api/upload -> raw PDF body; runs the quber workflow (fuse -> figure -> ingest) as a job GET /api/jobs/{job_id} -> job stage (uploaded/waiting/extracting/ reading figures/ingesting/done/failed) plus a tail of its log POST /api/batch -> start a run of many questions -> job id GET /api/batch/{id} -> the run's state and rows finished so far GET /api/batch/{id}/events -> SSE, one event per question completing GET /api/batch/{id}/export.xlsx and .csv -> the run as a workbook / text GET /healthz -> readiness: 200 once the database answers and the embedding model is loaded GET /static/{path} -> the UI's assets `quber.playground.auth.install` adds `/api/me`, and when sign-in is on also `/login`, `/callback` and `/logout`. Document identity is the integer id plus the content hash. The storage key (`doc_key`, the hash's first 16 hex chars) names the staged files and is what ingest subprocesses take. The document list leaves it out, but the PATCH response returns the full document row and so carries it. Labels are derived from the filing metadata at read time — see `quber.playground.metadata`. Uploads take the PDF as the raw request body (no multipart dependency). The bytes are hashed before anything runs: the same bytes as an existing document are rejected outright (409, kind "duplicate"), and a filing-tuple match is refused (409, kind "tuple") unless the request carries the replace intent — then the swap happens when the new ingest completes. An s3:// source cannot be hashed before its job stages it, so those checks run inside the job and a failure carries the same structured `error_info`. Every upload is admitted; a semaphore bounds how many run their pipelines at once (`settings.playground.upload_concurrency`), and a job holding no slot yet reports the `waiting` stage rather than appearing stalled on its first step. Run: uv run quber playground ) annotationsN)Path)AnyDictListLiteral LiteralStringOptional)FastAPI HTTPExceptionRequest) FileResponse HTMLResponseResponseStreamingResponse)logger) BaseModel) fetch_file)authdb embeddingexportgpujobsmetadatastorage)DocumentIdentity)answer) answer_stream) BatchRunnerRunStateparse_questions)RetrievedChunkretrieve) document_key) get_settingsstaticuploadszQuber Playground)titlestartupcTSSjn[R"USS9R5 g)zLoad the embedding model in the background at startup. It otherwise loads lazily on the first question, costing that question ~5 seconds.cX[R"S5 [R5 g)Nzwarm up)r embed_query_EMBEDDING_READYset2/home/mande/repo/quber/src/quber/playground/app.pywarm"warm_embedding_model..warmesi(r2T)targetdaemonNreturnNone) threadingThreadstart)r4s r3warm_embedding_modelr>`s" D.446r2c[R"5 [R"S[R[ S9 g)zjThe job tables exist before anything writes to them, and this process starts beating for the jobs it runs.c[R5Vs/sH oS[R;dMUSPM" sn$s snf)Nstagejob_id)JOBSvaluesrTERMINAL_STAGES)js r3"open_job_records..rs2iAG*TXThThBhKAhKiis A A) upload_idsrun_idsstoreN)r ensure_tablesstart_heartbeatBATCH running_ids BATCH_STOREr1r2r3open_job_recordsrQls- i!!r2z/healthzF)include_in_schemac [R5(d [SSSS9$[R"5nUR S5R 5 SSS5 [S SS 9$!,(df  N=f![aAn[[R"SS[U5R305SSS9sSnA$SnAff=f) a Readiness, not liveness: 200 only once the database answers and the embedding model is loaded, 503 before. The load balancer's target group and the scale-to-zero startup page both poll this, so a 200 means the app can take a question, not merely that the process is up.z${"status":"loading embedding model"}izapplication/json) status_code media_typezSELECT 1Nstatuszdatabase unreachable: z{"status":"ok"}rU) r/is_setrrconnectexecutefetchone Exceptionjsondumpstype__name__)connexcs r3healthzrcxs  " " $ $ 2Pb   ZZ\T LL $ - - / %2D EE\   JJ$:49;M;M:N"OP Q)   s:A> A-A>- A;7A>;A>> C 6C>C C cF\rSrSr%S\S'S\S'SrS\S'SrS \S 'S rg ) ChatRequestintdoc_idstrquestion kauto Literal['auto', 'value', 'text']wantr1Nr` __module__ __qualname____firstlineno____annotations__rlro__static_attributes__r1r2r3reres" KM AsK.4D *3r2rec\rSrSr%S\S'S\S'S\S'S\S 'S\S 'S rS\S 'S rS\S 'S rS\S'S rS\S'S r S\S'Sr S\S'S r S\S'S r S\S'S r S\S'S rS\S'Srg ) Referenceriref_id Optional[str]ref_typergpageOptional[dict]bboxlabelNrVnotetext Optional[int]rowcolFboolflaggedchartsegmentreasonprintedr1)r`rqrrrsrtrVrrrrrrrrrrur1r2r3rwrws K I  J!FM D-D-CCGTE=!G]!!FM !G]!r2rwc>\rSrSr%S\S'S\S'S\S'S\S'S rg ) RetrievedInforichunk_id chunk_typergr|floatscorer1N)r`rqrrrsrtrur1r2r3rrsMO I Lr2rcP\rSrSr%S\S'S\S'S\S'SrS\S 'S rS \S 'S rg ) ChatResponserirList[Reference] referenceszList[RetrievedInfo] retrievedproseshapeNr}valuer1)r`rqrrrsrtrrrur1r2r3rrs) K"" E3 E> r2rc2SSKJn [SU55$)NrCELL_STATUS_REFERENCEc3^# UH#oR(dMURv M% g7fN)inspectcode).0ss r3 !_inspect_codes..sH%:iiVQVV%:s--)quber.core.extractors.baser frozensetrs r3_inspect_codesrs@ H%:H HHr2cfSSKJn UVs0sHoRUR_M sn$s snf)Nrr)rrrr)rrs r3_status_reasonsrs)@%: ;%:FFAGGO%: ;; ;s .zprints '([^']*)'cU(dgSnUS:Xa4U(a-[RU5nU(aURS5nU(aU(a SUSUS3U4$[R X5U4$)zMThe plain-language line for a status, and the printed figure a misread names.)NNN value_misreadzThe page prints z here, not z. Check which is right.)_PRINTED_IN_NOTEsearchgroupSTATUS_REASONSget)rVrservedrms r3_reasonrsp G  T  # #D ) ggajG6!'+fX=TUW^^^   f -w 66r2cUS:XaSUS3$US;aSUS3$US:XaSUS3$US:XaSUS 3$US :XaSUS 3$SUS 3$) zWhat the source is, in the reader's words. The status never rides the label: it has its own field, and its plain-language line is `reason`. tableCellzPage z , table cell)table chunkTablez, table figureValuez , chart value line_itemz , table rowpicturez, figurez, textr1)r{page1rVs r3_labelrs;ug\****ugW%%= ug]++;ug[))9ugX&& 5'  r2c[S- R5nSHZn[R"[U- R 55R 5SSnUR SU3SUSU35nM\ U$)uThe index shell with content-hashed asset links. A browser that cached a previous deploy of app.js kept serving it on a plain reload, because the asset URLs never change. Stamping each mutable asset with a hash of its current bytes makes any change a new URL; the pinned vendor files stay as they are. Hashing runs per request — the files are small and the reload server would otherwise need cache invalidation of its own. z index.html)z styles.csszapp.csszapp.jsNz/static/z?v=)STATIC read_texthashlibsha256 read_bytes hexdigestreplace)htmlnamedigests r3_stamped_indexrsw \ ! , , .D3$ : : <=GGI"1M||htf-$s6(/KL4 Kr2/)response_classc*[[5SS0S9$)Nz Cache-Controlzno-cache)headers)rrr1r2r3indexr s (?J2O PPr2z/static/{name:path}c[U- nUR5(a6UR5R[R55(d[ SSU35e[ U5$)Nz no asset )ris_fileresolveis_relative_tor r)rpaths r3 static_assetrsT D=D <<>>!>!>v~~?O!P!PC9TF!344  r2zwd.id, d.doc_key, d.filename, d.page_count, d.ade_version, d.title, d.folder, d.filing_type, d.year, d.period, d.versionr _DOC_COLUMNSc USUSUSUSUSS:XaSOSUS US US US US USS. n[R"US US US US5US'U$)Nrrr)z quber-fusionfusionader rk) iddoc_keyfilename page_countformatr*folder filing_typeyearperiodversionr)r filing_label)rdocs r3 _doc_dictr)sdQ4aDddn4(%1A$t!A$R5 C((1qtQqT1R5ACL Jr2c[R"5nURS[-S-U45R 5nSSS5 Wc[ SSU35e[ U5$!,(df  N+=f)NSELECT z0 FROM ade_playground.documents d WHERE d.id = %sr no document )rrYrZrr[r r)rhrars r3_doc_rowr;sl  LL  $'Y Y I  (*    yC<x!899 Q< s ,A'' A5z/api/documentscd[R"5nURS[-S-[ [ 545R 5nSSS5 /nWH*n[U5nUS USUS'URU5 M, [R"U5 U$!,(df  NX=f)Nrz, COUNT(g.id) FILTER (WHERE g.status = ANY(%s)) AS flags FROM ade_playground.documents d LEFT JOIN ade_playground.groundings g ON g.document_id = d.id GROUP BY d.id ORDER BY lower(coalesce(d.title, d.filename))r flags) rrYrZrlist INSPECT_CODESfetchallrappendrannotate_overlaps)rarowsdocsrrs r3 documentsrFs ||  N N - "  (*   D l  NuG  C   t$ K% s 9B!! B/cn\rSrSr%SrSrS\S'SrS\S'SrS\S'Sr S\S'Sr S\S 'Sr S\S 'S r g) DocumentPatchi`u[The editable metadata: presentation (title, folder) and the filing fields. Edits change organization and labels only — never identity, and never blocked by collisions; two documents landing on one tuple simply both carry the possible-duplicate flag. Fields arrive as strings so an emptied input clears its column; year and version parse to ints.Nrzr*rrrrrr1) r`rqrrrs__doc__r*rtrrrrrrur1r2r3rr`sFH  E= FM !%K%D- FM !G]!r2rc[U5nX$s=::aU::dO [SUSUSU35eU$![a [SUS35Sef=f)Nz must be a numberz must be between z and )rg ValueErrorr )rrawlohirs r3 _parse_introslGC  " CD6):2$eB4!HII L GCD6):!;<$FGs 1A z/api/documents/{doc_id}cLURb+URR5(d [SS5e/n/nSUR4SUR4SUR4SUR 4/nUHFupVUcM UR US-5 UR UR5=(d S5 MH URbYUR S 5 UR URR5(a[S URS S 5OS5 URbYUR S 5 UR URR5(a[SURSS5OS5 U(d [SS5eSSRU5-S-n[R"5nURU/UQUP75R5n SSS5 W c[SSU35e[U5$!,(df  N+=f)zUpdate a document's metadata. The ingested artifacts are untouched; an emptied field clears its column (folder falls back under the ungrouped heading, filing fields drop out of the label).Nrztitle cannot be emptyr*rrrz = %sz year = %srili4z version = %srriznothing to updatez$UPDATE ade_playground.documents SET z, z WHERE id = %s RETURNING idrr)r*stripr rrrrrrrjoinrrYrZr[r) rhpatchsetsparams text_fieldscolumnrqueryrars r3patch_documentrys  {{u{{'8'8':':C!899 "DF %++ 5<< ))* 5<< >K #  ? KK( ) MM#))+- .# zz K  EJJDTDTDVDVjT4@\`a }}  N# emmFYFYF[F[jEMM1cBaef C!455. ))D/  ' ( ll5"3F"3F"34==?  {C<x!899 F  s &H H#c[R"5H1nURS5U:Xa gURS5U:XdM1 g [R U5(agg)zWhy this document cannot be pulled out from under running work, or None. An ingest can touch it two ways: a replace targeting it, or an s3-sourced job that will resolve to the same bytes and key. replace_idzIa replacement for this document is still ingesting; wait for it to finishrzBan ingest of this document is still running; wait for it to finishzGa batch run against this document is still going; wait for it to finishN)r live_uploadsrrN running_for)rhrjobs r3 _ingest_busyrsX  " 77< F *^ 779  (W #    X r2cSnUVs/sHn[UU3- PM nnUR[US3- 5 U$s snf)a'The staged files a document's storage key names: the served PDF and the cached parse and figure JSONs in the data directory, and the uploaded original. The pipeline workdir is not in the list; `_remove_doc_files` removes it separately. The `uploads/.source.json` manifest is in neither.) .pdfz .ade.json .unified.json .tables.json .fusion.json.scanned-tables.json .figures.json.figure-values.jsonz .charts.jsonr)DATA_DIRrUPLOADS)rsuffixesrpathss r3 _doc_filesr%sN H2: :AX7)A3 'E : LLgYd++, L ;s=cSn[U5H/nUR5(dMUR5 US- nM1 [US3- nUR5(a[R "U5 US- nU$)Nrr -artifacts)r%existsunlinkr"shutilrmtree)rremovedr artifactss r3_remove_doc_filesr.soG7# ;;== KKM qLG$WIZ00I i 1  Nr2c [U5n[XS5nU(a [SU5e[R"5nUR SU45R 5nUR SU45 SSS5 Wce[US5n[R"SUUS=(d USUS US US 9 XS US US .$!,(df  N[=f) uRemove a document outright: the row (chunks, groundings, vectors and flags cascade with it), the staged files, and the upload artifacts. A hard delete, documented as such: there is no tombstone, and the cascade plus the file removal leave nothing for a re-ingest to resurrect. Refused with 409 while an ingest touching the document or a batch run against it is still going — the server owns no cancellation, so the work finishes or fails before the ground moves. raSELECT COUNT(DISTINCT c.id), COUNT(DISTINCT g.id) FROM ade_playground.documents d LEFT JOIN ade_playground.chunks c ON c.document_id = d.id LEFT JOIN ade_playground.groundings g ON g.document_id = d.id WHERE d.id = %s2DELETE FROM ade_playground.documents WHERE id = %sNzZdocument removed: {id} '{title}' ({chunks} chunks, {groundings} groundings, {files} files)r*rrr)rr*chunks groundingsfiles)r,r2r3r4) rrr rrYrZr[r.rinfo)rhrbusyracounts removed_filess r3delete_documentr9s 6 C I /D C&&  " I   (*  IF9U    %c)n5M KKd 'l-c*oay!9 &)Vc dd) s 5C  Cz/api/document/{doc_id}/pdfc[U5n[R"US5nUc[SSU35e[ USS9$)NrrzNo PDF staged for document zapplication/pdfrW)rr local_pdfr r)rhrrs r3pdfr<sH 6 C   S^ ,D |C#>vh!GHH ): ;;r2c\[U[5(aU$[R"U5$)zCA groundings bbox as a dict, whether the driver returned it parsed.) isinstancedictr]loads)rs r3 _bbox_dictrAs!S$''3^^TST;dTScg[TS5m[UU4SjT55$)Nrr)Fc3p># UH+upUTS=(d S:H=(a [TU5v M- g7f)rrN)rJ)rr|finerGrs r3r5_resolve_refs..buries_fine..Cs1cXb*$4AaDIA&E=+EEXbs36)rAany)rrG fine_boxes fine_typess`@r3 buries_fine"_resolve_refs..buries_fine?s: Q4: 11Q4 cXbcccr2rrrrrrrr)ryr{r|r~rrVrrrrrrrrr)rtupler9r)rrYrZrr0rDr>r?r]r@raddsplitrArrrwrrr)rh cited_idsrarrby_idcovered_picturespositionrL covered_linesrrUrefscidrr~rVr cell_textrrrSrTs @@r3 _resolve_refsrbs  || B   (*  # #dqT1WdE #u \\^ Q4= AaDL %adD111Q4AaD 1Q48HVX~2**=9 ;   - EM \\^ Q4; !A$, %adD111Q4AaD 1Q48HVX~2""5) ?   1C!3A!6 7vcUC D  -J161?AQ4:CU'Z[\]Z^'!A$)!j1&'Jdd D IIcN  Q49 1)9!9  Q4; 1Q4=#8  q>> 1a!!A$--qt!DJJqt4DRV"#A$!adi%adD111Q4AaD 1Q48HVZ!&$ : t1QqT5&1^((/^((/V?-(?@~2**73!R,,W5  D Kk  $>s#"O(O: O?-O?5 O?( O7cv\rSrSr%SrS\S'S\S'S\S'S \S 'S\S 'S rS\S 'SrS\S'S rS\S'Sr g )FlagikzOne item of the document's review queue: a cell whose provenance status is registered for inspection, or a table-level dropped-text flag.riryrzr{rgr|r}r~rVNrrrr1) r`rqrrrsrrtrrrrur1r2r3rdrdksBP K I  KD-D#N FM r2rdz/api/document/{doc_id}/flags)response_modelc[R"5nURSU[[545R 5nSSS5 WVs/sHn[ USUSUS=(d SS-[US[5(aUSO$US(a[R"US5OSUSUSUS =(d S [USUSUS 5SS 9PM sn$!,(df  N=fs snf) z3Every flagged item for one document, in page order.zSELECT g.ref_id, g.ref_type, g.page, g.bbox, g.status, g.note, g.cell_text FROM ade_playground.groundings g WHERE g.document_id = %s AND g.status = ANY(%s) ORDER BY g.page, g.ref_idNrrrr)rrrre)ryr{r|r~rVrrr) rrYrZrrrrdr>r?r]r@r)rhrarrs r3document_flagsrhys || ,T-( )  (*  & A Q4qTA$)!q#AaD$//1!A$djj16FTXQ4111Q41qt,Q/     s0C-BC>- C;zdict[str, dict]rCr)uploadedwaiting extractingreading figures ingestingdpt-3grouped)modelchunkingdpt-2flat)rorsz10-Qz10-KcURS5=(d SR5R5nU(aU[;aSOSnUSU34$SSKnSSKnSSKJn SS KJ n URU5n U Rn SSS5 X!RS 3- n U"US S U 5 URU"5RU R5W 55n U R!S S9 U R"S:XaSSU R$34$SSU R$34$!,(df  N=f![&a(n SS[)U 5R*SU S34sSn A $Sn A ff=f)uThe figure scan's page breadth for one upload, and the reason for it. Typed documents are decided by their filing type alone. An untyped document is never allowed to default to full coverage — the type field is optional and routinely left blank, and a blank must not buy an 80-page scan by accident. A cover attribution agent reads the first page instead and decides presentation (all pages) against prose filing (nominated only); any failure in that read falls back to nominated-only, with the fallback stated in the returned reason so the record shows breadth was reduced and why. The verdict decides breadth only — it is never written to the document's filing type, which stays exactly as the user entered it. rre nominatedallz filing type rN)get_cover_attributor) render_pagez .cover.pngrdT missing_ok presentationzattributed presentation: zattributed filing: zattribution failed (: z); nominated-only fallback)rr upperNOMINATED_ONLY_TYPESasynciofitzquber.agents.cover_attributionrxquber.files.pdfryopenrstemrun attributerr)kindrr\r_r`)rr<workdirrscoperrrxryrrcoververdictrbs r3 _scan_breadthrsQ77=)/R668>>@K*.BB  [M222C+ YYs^sJXXJj11CC'++24>>u?O?O?QS]^_  % <<> )5gnn5EFF F1'..1ABBB^  "49#5#5"6b=W X   s=(D09 DBD0D0 D-)D00 E":EE"E"c\rSrSr%S\S'S\S'SrS\S'SrS\S'S\S '\S rS \S 'Sr S\S 'Sr S\S'Sr S\S'S \S'Sr g) JobStatusirirBrNrzr*rrAr List[str]stageserrorr} error_inforrhlogr1) r`rqrrrsrtr*rPIPELINE_STAGESrrrrhrur1r2r3rrs[ KME= FM J'1FI1E= "&J% FM Nr2rcXS'USRSUSSRU535 [R"U5 [R "U[ [R[RSS9nURceURHJnUSRUR55 [US5[:dM=USS[*2 ML UR5 [R"US[ USS 3- 5 [R"U5 UR"S :wa[%US UR"35eg) NrArz--- r~ T)cwdstdoutstderrrrr'rz exited with code )rr r save_upload subprocessPopen REPO_ROOTPIPESTDOUTrrstriplen _LOG_TAILwaitrpublish_workdirr" returncode RuntimeError)rrAcmdproclines r3 _run_stagers&LJUG2chhsm_56S   CYzzO`O`gk lD ;; "" "  E $++-( s5z?Y &E ;YJ;' IIK C NGY8H 6S,STS !eW$6t6GHIIr2c>^TSnSU4SjjnSTS'[R"T5 [TSSSSSUS [U5S S / 5 [R "X2U5 [R"T5 [TSSSSS US [U5S [U5/ 5 g)aThe extracting stage where this host has no GPU, in three steps run one after another: the table engine runs here as `quber table`, then the docling parse goes to the worker and is waited on, then `quber fuse` reads the two sets of artifacts from the workdir.rcv>TSRU5 [TS5[:aTSS[*2 gg)Nr)rrr)rrs r3r#_run_hosted_extraction..logs< E $ s5z?Y &E ;YJ;' 'r2rlrAuvrquberr-oz --llm-backendapifusez--artifacts-dirN)rrir9r:)rrrrirparse)rr<rrrs` r3_run_hosted_extractionrs )nG(  CLS \D%'3c'lTcejkIIg$S  ugvs,=s7|TSVW^S_`r2cbSUS'[R"U5 [R5 g)zHold until a pipeline slot frees. Admitted-but-waiting is its own stage: a queued document has something honest to show rather than appearing stalled on its first step.rkrAN)rr _UPLOAD_SLOTSacquire)rs r3 _await_slotr,s'CLSr2c ^^ [R"5nURSTS45R5nSSS5 W(a7US=(d USnSSUSUS =(d S S 3US USUS S .S.$[R "5Vs/sHoDSTS:wdMUPM nn[ U4SjU55(aSSS.$[R"TSTSTSTSTS5m T cgUHun[R"URS5URS5URS5URS5URS55nUT :XdMpSSS.s $ [R"5nURS[-S-5R5nSSS5 [U 4SjSW55S5nUcgTS(d)SSUS S!US=(d S S 3US"US#USS .S.$[US"US$5n U (a S%S&U S 3S.$US"TS''US$TS('g!,(df  GN=fs snf!,(df  N=f))u|The reason this upload must not run, as the structured error_info the modal renders — or None. Checks, in order: the same bytes already in the library (or already being ingested), then the filing tuple against existing documents and in-flight jobs. A tuple match with the replace intent records the target on the job instead of refusing, unless the target is under running work.zXSELECT id, title, folder, filename FROM ade_playground.documents WHERE content_hash = %s content_hashNrr) duplicateu/This exact file is already in the library — 'z' in r Ungrouped.r)rr*r)rmessageexistingrBc3P># UHoRS5TS:Hv M g7f)rN)r)rrFrs r3r#_admission_error..Is! >v!55 s9~ -vs#&z*This exact file is already being ingested.)rrrrrrrrWz?A document with this filing metadata is already being ingested.rz FROM ade_playground.documents dc 3># UH6n[R"USUSUSUSUS5T:XdM2Uv M8 g7f)rrrrrN)r tuple_key)rdkeys r3rr[sN 1!!!H+q/?6AhKYZ[dYefjmm A1s 1A Ac38# UHn[U5v M g7fr)r)rrs r3rr]s1DqillDsrzA rz already exists in rr*rr6zCannot replace: r replace_key)rrYrZr[rrrRrrrrrnextr) rradupr*rFothers other_keyrmatchr6rs ` @r3_admission_errorr5s ll f  "  (*    A #a&HuUXYZU[Uj_jTkklm"1vA#a&I  **, M,A( s8}0La,F M >v >>>#0\]]   S]C ,>F SQY]\_`i\j kC { && EE(OQUU=1155=!%%/STSXSXYbSc   \   ||I 47YYZcce   1D1  E } y>E'N++>uX?]R]>^^_`$T{U7^uU]_  d U9%5 6D -=dV1+EFFd C y)C  g N s#%I I-I*I! I! I/cSn[R"5nUR5 URSUR S5UR S5UR S5UR S5UR S5UR S5UR S 5US 45 UR S 5bY[ R US 5(aUS RS 5 O'URSUS 45 UR S5nSSS5 URSUS 45R5nSSS5 U(a$[U5 [R"SUS US9 W(aUSOSUS'[R"U5 g!,(df  N=f!,(df  Nr=f)zStamp the upload's metadata onto the freshly ingested row and, for a replace, swap the target out in the same transaction. Ingest recreates the document row, so this runs after it.NzUPDATE ade_playground.documents SET title = %s, folder = %s, filing_type = %s, year = %s, period = %s, version = %s, track = %s WHERE doc_key = %sr*rrrrrtrackrrrzxreplace target is under a running batch; both documents kept (remove the old one from the library when the run finishes)r1rz:SELECT id FROM ade_playground.documents WHERE doc_key = %szreplaced document {id} ({key}))rrrrh)rrY transactionrZrrNrrr[r.rr5rr)r replaced_keyrars r3 _finalize_jobrrs#'L      LL) GGG$GGH%GGM*GGFOGGH%GGI&GGG$ N    ww|$0$$S%677J%%V LL!UX[\hXiWkl#&77=#9L9 :ll H3y>J[ (* = B,' 4\9JP\]!CFtCMSI  s#F3C2F",F3" F0 ,F33 GcUcUce[SUSS3- n[X#5 UR5n[R "U5R 5US 'US SS US '[U5nUb!URS S9 SUS'USUS 'XPS'gUR[US S3- 5 [[US S3- 5$![anSUS'SUSU3US 'SnAgSnAff=f)aeResolve the upload to local bytes, hash them, and run admission. Returns the staged PDF path to run the pipeline on, or None after marking the job failed. Each upload is admitted once. A body upload was hashed and admitted in the route, so it passes straight through. An s3 source is fetched, hashed and admitted here, because the route never had its bytes.Nztmp-rBrfailedrAzcould not fetch r~rrrTr{rr) r"rrr\rrrrr)renameri)r pdf_bytess3_uristagedrbr5s r3_admit_or_failrs'!!!T#h-55  v &))+I &nnY7AACN^,Sb1I$   MMTM *#CL ?CL $  g3y>"2$ 778 wC N+400 11 #CL-fXRu=CL sC C0C++C0cj[Un[U5 [X1U5nUc,[R"U5 [ R 5 gUSn[US3- nURSSS9 Un[R"XSSUSUS9n[R"U[U5U/5 [URS 5=(d [n [!U[U5U5upXS 'XS 'X#S '[R"U5 ["R$"S X U 5 [&R("5(a [+X4U5 O[-USSSSSUS[/U5/5 [-USSSSSUS[/XgS3- 5S[/U5SU SSU / 5 [-USSSSSSSUS USS![/XgS3- 5S"[/XgS#3- 5S$[/XgS%3- 5S&[/XgS'3- 5S([/XgS)3- 5S*[/XgS+3- 5S,US-U S./5 [1U5 S/US0'[R"U5 [ R 5 g![2an S1US0'[/U 5US2'Sn A NMSn A ff=f![R"U5 [ R 5 f=f)3aThe quber workflow: fuse, then read the figures, then ingest. The figure stage is part of the workflow rather than an option on it. Without it a document arrives holding only what the text layer gave up: a chart's plotted values are absent, a table printed as an image stands as the parse's own reading of it, and the notes that qualify a figure are attached to nothing. The standalone ADE path is a different pipeline and is unaffected. The stage runs on every upload, and it spends: each page it scans is paid for. `_scan_breadth` decides which pages. At nominated breadth only the pages the parse nominates are scanned, so a document with no nominated page costs nothing. At all-pages breadth every page is scanned. An artifact file the run did not write, such as the scanned tables of a document with no table captured, the ingest reads as "the run produced none of that kind" rather than as an error. Nrr'Tparentsexist_okrr)rr source_urir scan_pagesscan_pages_reasonrz&Upload {}: figure scan breadth {} ({})rlrrrrrrmfigurez--parserz--modelrqz--pagesrnpythonz-mzquber.playground.ingest_fusionz --doc-keyz --filenamez--unified-jsonz --tables-jsonrz --fusion-jsonrz--scanned-tables-jsonrz--figures-jsonrz--figure-values-jsonr z--pdfz --chunkingrrdonerArr)rCrrrrrreleaser"mkdirrwrite_source_manifestpublishrTRACKSr DEFAULT_TRACKrrr5r configuredrrrirr\) rBrrrr<rrbasemanifestrpagesbreadth_reasonrbs r3_run_jobrs" v,CV CSV4 ; b ci.wiz22 dT 200 *oC>   "3W 5 sL4dTWX_T`*a b  Gm445G g  &   0J Gm445Gl334Gl334'G&:;;< Gm445&G&9::;j!3  > cG   G 3xG   s0IG I J&I>9J>JJ,J2z /api/uploadc P# U [;a[SS[[535eSn UbGURS5(a$UR 5R S5(d [SS5eO:UR 5IShvN n U RS5(d [SS5e[R"5RSS n 0S U _S U_S U_S U_SU_SU_SU_SU_SU _SU _SS_SS_SS_SS_SS_SS_S[S_SS/S.En U b[R"U 5R5U S'U SSSU S'[U 5nUb [SU5e[R!S S S!9 [U SS3- R#U 5 O[R!S S S!9 [$ U [&U 'SSS5 [(R*"U 5 [,R."[0XU4S S"9R35 [5S$00U ES#/0ED6$GN!,(df  Nb=f7f)%ueRun the fusion pipeline on a PDF sent as the raw request body, or on an s3:// object named by `s3_uri` (body then stays empty). `track` selects the model line the document is processed on — see TRACKS. `filename` is the original name, kept for presentation and never for identity. The metadata fields are how the library presents and keys the document; this route and the PATCH are the only places they are ever set. `replace` carries the modal's confirmed intent to swap out the document whose filing tuple this upload matches. Every admitted upload waits in the `waiting` stage until a pipeline slot frees. rztrack must be one of Nzs3://rz*s3_uri must be an s3:// URI ending in .pdfs%PDFzrequest body is not a PDF rBrr*rrrrrrrrrrrrhrArjrr)rrrrr0Tr)r6argsr7rr1)rr sorted startswithlowerendswithbodyuuiduuid4hexrrrrrr"r write_bytes _JOBS_LOCKrCrrr;r<rr=r)requestrrr*rrrrrrrrrBrr5s r3uploadr(s4 FC#88H!IJJ D   ))1H1H1P1P%QR R2Q\\^#w''%@A A ZZ\  cr "F&H  &  {    & 7 7  d t 4  $ !" /(+#$)C, &nnT2<<>N^,Sb1I$  T* * dT 2 c)n%T* *77= dT 2 V  S HF&+A$OUUW  *)#)ub) **W$N s,BH&H D&H&/ H9AH& H#H&z/api/jobs/{job_id}c [RU5=(d [R"U5nU(d[ SSU35e[ S00UESUSU*S0ED6$)zThe job as this process holds it when it is running it, else as the database records it, which is how a poll lands on any task.rzno job rNr1)rCrr load_uploadr r)rBtailrs r3 job_statusrwsa ((6  6d..v6C C76(!344  :9#9uc%j$&89 ::r2cURS:Xa UR$URS:Xa UR$URS:Xa UR$[R "UR SS9SS9$)zWhat to put in `answer` for a consumer that only reads text. A figure renders as the figure, so the text field of a value-seeking answer is the value and not a sentence about it. r unanswerablescalarr]moder)indent)rrrrr]r^ model_dump)payloads r3 _readablerse ||w||||~%~~||x}} ::g((f(5a @@r2c[USS5nU(d [U5$U/UVs/sH o3U:wdM UPM sn-$s snf)uThe cited ids with the answer's own source leading. A value payload names the single cell or chunk it was read from in `source_id`. The UI activates the first reference that carries a box, so whatever id leads this list becomes the default highlight — and the value's own cell is the highlight the answer earned, not the wider line record the model happened to cite first. Payloads without a source (prose, unanswerable, series) keep the model's citation order. source_idN)getattrr)r rZrr`s r3cited_ids_source_firstrsCd3I I ;G#Y6F#G GGGs >>cXURn[[U5URUR SS9[ U[ X1R55UVs/sHDn[URURURS-[URS5S9PMF snS9$s snf)Nr]r rr)rrr|r)rrrrr)r rrrr rbrrZrrrr|roundr)rhresultr2r cs r3_chat_responsersnnG !ll  f - )?IYIY)Z[     166A:UZ[\[b[bdeUf      sA B'z /api/chatc# [UR5n[R"US5 [ USUR UR S9IShvN nU(d[SSURS35e[UR X R[R"U55IShvN n[URX25$NxN7f)NrrlrNo chunks for document  — is it ingested?) rrhr%r0r$rjrlr agent_answerrorfrom_rowr)reqrr2rs r3chatrs 3:: CS^$C NCLLCEEB BF C#:3::,FZ![\\ fhh@P@Y@YZ]@^_ _F #**f 55 C`s%ACCAC4C5CCz/api/chat/streamc`^^^# [TR5n[R"US5 [ USTR TR S9IShvN mT(d[SSTRS35e[R"U5mUUU4Sjn[U"5SS 9$NR7f) zSame contract as /api/chat, streamed: server-sent events carrying the answer text as it generates ('delta'), then the full response ('done').rrNrrrc># [TRTTRT5ShvN upUS:Xa"S[R"SUS.5S37v M3[ TR UT5R5nS[R"SUS.5S37v MzNu g![a3nS[R"S[U5S.5S37v SnAgSnAff=f7f) Ndeltadata: )r_r r)r_r r)r_detail) agent_answer_streamrjror]r^rrhr r\ri)rr rrbr2identityrs r3eventschat_stream..eventss S':3<<QTQYQY[c'd Wmd7?"4::w'.R#S"TTXYY)#**gvFQQSD"4::v$.O#P"QQUVV  W'd  S4::w#c(&KLMTR R R SsOC%"B%B#B!B#A2B%!B##B%$C%% C"/)CC%C""C%text/event-streamrW) rrhr%r0r$rjrlr rrr)rrr(r2r's` @@r3 chat_streamr+s 3:: CS^$C NCLLCEEB BF C#:3::,FZ![\\((-H S VX2E FF#CsAB.B,AB.)rKcF\rSrSr%S\S'S\S'SrS\S'SrS \S 'S rg ) BatchRequestirgrhri questionsrkrlrmrnror1Nrpr1r2r3r-r-s" KN AsK-3D *3r2r-z /api/batchc^^^# [TR5nU(d [SS5e[TR5nUSm[ R "T5 [TUSSS9IShvN nU(d[SS TRS 35e[R"U5mS UU4S jjnSUU4S jjn[RTRUTRXE5$Ny7f)zStart a run. The work continues if the page is closed; the page keeps the returned job id in sessionStorage and re-attaches through the state and events routes.ru4no questions — the batch window takes one per linerrrrNrrrcJ># [TUTRS9IShvN $N7f)Nr)r$rl)rjrrs r3 retrieve_one!batch_start..retrieve_onesgx3559999s #!#c># [XTRT5IShvN n[TRX!5R SS9$N'7f)Nr]r )rrorrhr )rjr2rr'rs r3 answer_onebatch_start..answer_ones@#HchhIIcjj&9DD&DQQJsAA(A)rjrir9List[RetrievedChunk])rjrir2r6r9Dict[str, Any]) r"r.r rrhr%r0r$rrrNr=ro)rr.rprober1r4rr's` @@r3 batch_startr9s   .I C!WXX 3:: C)nGW7IaLA6 6E C#:3::,FZ![\\((-H::RR ;;szz9chh QQ 7sA*C+/C)0A:C+cV[RU5nUc[SSUS35eU$)Nrzno run u — the server no longer has it)rNrr rBstates r3 _run_stater=s3 IIf E }C76(2R!STT Lr2z/api/batch/{job_id}c[U5$r)r=)rBs r3 batch_stater? s f r2z/api/batch/{job_id}/eventscL^# [T5 U4Sjn[U"5SS9$7f)uOne server-sent event per question completing — index, total, outcome — then a final `done` event. Watching a run and re-attaching to one are different needs; this is the watching route.c># [RT5ShvN nS[R"U5S37v M(N# g7f)Nr#r$)rNr(r]r^)eventrBs r3streambatch_events..streams: <</ 3%4::e,-T2 2 3/s%AAAA AAAr*rW)r=r)rBrCs` r3 batch_eventsrEs' v3 VX2E FFs!$c[UR5n[R"USUSUS5$![a gf=f)zThe export filename's document part, down the label precedence. A run whose document was removed mid-session still exports; it just gets the generic stem.documentrr*r)rrhr r export_stem)r<rs r3 _export_stemrIsLu||$   G c'lC O LL s8 AAz/api/batch/{job_id}/export.xlsxc z[U5n[[R"U5SSS[ U5SUS30S9$)NzAapplication/vnd.openxmlformats-officedocument.spreadsheetml.sheetContent-Dispositionattachment; filename="batch-rMz.xlsx"rUr)r=rr write_xlsxrIr;s r3batch_export_xlsxrO*sM v E % V&*F|TYGZF[[\]c\ddj(kl r2z/api/batch/{job_id}/export.csvc z[U5n[[R"U5SSS[ U5SUS30S9$)Nztext/csv; charset=utf-8rKrLrMz.csv"rM)r=rr write_csvrIr;s r3batch_export_csvrR4sM v E ,&*F|TYGZF[[\]c\ddi(jk r2r8)r9r)r9zfrozenset[str])r9zDict[str, str])rVrzrrzrrzr9z#tuple[Optional[str], Optional[str]]r)r{rzrrgrVrzr9ri)r9ri)r9r)rrir9r)rz'tuple[Any, ...]'r9r7)rhrgr9r7)r9z list[dict]) rrirrirrgrrgr9rg)rhrgr rr9r?)rhrgrrir9rz)rrir9z List[Path])rrir9rg)rhrgr9r?)rhrgr9r)rrr9r?)g{Gzt?)rGr?rHr?rIrr9r)rhrgrZrr9r)rhrgr9z List[Flag])rr7r<rrrr9ztuple[str, str])rr?rArirrr9r:)rr?r<rirrr9r:)rr?r9r:)rr?r9r})rr?rOptional[bytes]rrzr9rz)rBrirrSrrzr9r:)rr rrirrzr*rzrrzrrzrrrrzrrrrrrir9r)r)rBrirrgr9r)rZrr9r)rhrgr9r)rrer9r)rrer9r)rr-r9r!)rBrir9r!)rBrir9r)r<r!r9ri)rBrir9r)r __future__rrr]rer*rr;rpathlibrtypingrrrrr r fastapir r r fastapi.responsesrrrrlogururpydanticrquber.files.cacherquber.playgroundrrrrrrrr!quber.playground.answers.documentr$quber.playground.answers.expectationrrrr&quber.playground.batchr r!r"quber.playground.retrievalr#r$quber.playground.tracingr%quber.settingsr&__file__ with_namer playgrounddata_dirr!r"rrappinstall LOGIN_ENABLEDEventr/on_eventr>rQrrcrerwrrrrrrcompilerrrrrrrrtrrrrrr rrr%r.deleter9r<rArJrbrdrhrCLockrr Semaphoreupload_concurrencyrrrrrrrrrrrrrrpostrrrrrrr+ BatchStorerPrNr-r9r=r?rErIrOrRr1r2r3rts1f#  DD33UU(VVV>GUII?1' h ! !( + > $ $ - - Y  N " "1 %  &' S,.";";< ??$i77iu-F.F*4)4" "0I !9 !I <   "::12 7  7!. 78E 7( 7 ! "\*Q+Q   1 m$ 2 "I " $%#&#L ,  %&"e'"eJ %&<'<= Yx !9 ! 'T CD<o ^^   ##LN$=$=$P$PQ  SI 6F 3  '& R .J&4:z*Z2