j='jSrSSKJr SSKrSSKrSSKJr SSKJrJ r J r J r J r SSK Jr SSKrSSKrSSKJr SSKJr SS KJr S rS rS rS rS rSSSS.r\ \/S4rSSjrSSjr Sr!S Sjr"S!Sjr#S"Sjr$S#Sjr%S$Sjr&S%Sjr'S&Sjr(S'Sjr)\RT4S(Sjjr+g))aThe GPU stages of an upload, run on the RunPod worker. The hosted playground has no GPU. Two stages of an upload need one: the docling parse of the document, and the embedding of its chunks at ingestion. Both go to the RunPod serverless worker under the contract the inbound-drop Lambda already uses with it: the job payload carries presigned S3 URLs for everything the worker reads and writes, the worker holds no credentials, and a completion marker written strictly after the outputs says the outputs are whole. The files land in the document's own prefix in the extracts bucket, so a parse's artifacts and an embedding's vectors sit beside everything else the document owns. ``configured`` says whether this host sends work there at all. It needs RUNPOD_ENDPOINT_ID, RUNPOD_API_KEY and QUBER_PLAYGROUND_ARTIFACTS_URI all set. Without them, the app runs the docling parse on the local GPU in a `quber fuse` subprocess, and the chunks embed locally in the ingest process. ) annotationsN)Path)AnyCallableDictListOptional)urlparse)Config)logger) get_settingsi@8 i4docling confidencecells)documentrrc[5Rn[UR=(a UR=(a UR 5$N)r playgroundboolrunpod_endpoint_idrunpod_api_key artifacts_uri)ps 2/home/mande/repo/quber/src/quber/playground/gpu.py configuredr2s4!!A $$M)9)9Maoo NNc[5RR=(d Sn[U5nURUR R S5S-U-S-4$)N/)r rrr netlocpathstrip)doc_keyuriparseds r_bucket_and_prefixr(7sP . # # 1 1 7RC c]F ==&++++C036@3F FFrc[R"S[5RR=(d S[ SS9S9$)Ns3z us-east-1s3v4)signature_version) region_nameconfig)boto3clientr s3_cacheregionr rr_s3r4=s6 <<  N++22Ak/ rc@[5RXUS.[S9$)NBucketKey)Params ExpiresIn)r4generate_presigned_urlURL_EXPIRY_SECONDS)methodbucketkeys r_presignr@Hs' 5 ' '5AS ( rcL[5Rn[R"SURS3SU0SSUR 30SS9nUR 5 UR5n[R"SURS 5URS 55 US $) Nhttps://api.runpod.ai/v2/z/runinput AuthorizationBearer )jsonheaderstimeoutzrunpod job {} ({})idstatus) r rhttpxpostrrraise_for_statusrGr infoget)payloadrresponsejobs r_submitrTNs!!Azz #A$8$8#9>w  GA,<,<+=">? H   --/C KK$cggdmSWWX5FG t9rc[5nURXS9nUSR 5U:agUSR 5$!URRa gf=f)aThe object's bytes, or None when it does not exist or predates ``since``. Nothing under a document's prefix is ever deleted, so a previous run's marker and log stay where they are; a run tells its own apart by the object's write time against the moment it submitted the job.r6N LastModifiedBody)r4 get_object exceptions NoSuchKey timestampread)r>r?sincer*rRs r_read_if_freshr^\sm B===8))+e3 F  "" == " "sAA#"A#cP[XU5nUb[R"U5$S$r)r^rGloadsr>r?r]bodys r_read_json_if_freshrcls' &u -D#/4::d 9T9rcH[XU5nUbURSS5$S$)Nutf-8replace)r^decoderas r_read_text_if_freshrhqs) &u -D.2.>4;;w *HDHrc[5Rn[R"SURSU3SSUR 30SS9nUR 5 UR5$)NrBz/status/rDrErF)rHrI)r rrLrPrrrNrG)job_idrrRs r _job_statusrkvsg!!Ayy #A$8$8#9&J GA,<,<+=">?H   ==?rc [R"5nSnSn Sn [XU5n U bU(a7[XU5=(d Sn U R5USH n U"U 5 M U R S5S:wa1[ SU R SS 5S U R S S 535eU $[R"5U- nX:a[ S [U5SUS35e[U5nUR S5nUS;aR[ SUSUS [R"UR S 5=(d UR S55SS35eUS:XaxU =(d [R"5n [R"5U - S[-:a7[ SUS[R"UR S55SS35eU(aw[R"5U - [:aW[R"5n [XU5n U (a/U R5nUUSH n U"U 5 M [U5n[R"[5 GMZ)a!Poll until the marker exists, mirroring the worker's log into ``log`` as it grows. The marker is the truth about the outputs; the job's own status is watched beside it so a job that ends without writing one, or that RunPod fails or cancels, is reported at once rather than at the deadline.rgNr rKcompletezworker failed at stage?z: errorzno error recordedzno completion marker after zs ())FAILED CANCELLED TIMED_OUTz worker job  outputi  COMPLETEDz0 completed without a completion marker; output: )timercrh splitlinesrP RuntimeError TimeoutErrorintrkrGdumps POLL_SECONDSLOG_MIRROR_SECONDSlensleep)r> marker_keylog_keydeadlinelogrjr]startedmirrored last_mirrorended_atmarkertextlineelapsedrKstateliness r_wait_for_markerrs=iikGHK $H $V?  *6EBHb OO-hi8DI9zz(#z1"' 7C(@'AFJJwXkDlCmnM))+'  !@ tyy{[04FF))+K&v>D)!(),DI-u: < M rc[U5up4UUS3nUUS3n[5n[R"5S- nU[SX4US35[R 5V V s0sHupU [SX4USU S35_M sn n [SX55[SX65S .n [ U 5n U"S U 35 [X5U[X,U5n U"S U RS 5S U RS5SU RS5S35 [R5H.n URX4USU S3[XSU S3- 55 M0 gs sn n f)zParse the document on the worker and bring the three parse artifacts into ``workdir`` for fusion. The PDF must already be in the document's prefix as ``.pdf``.z.parse.complete.jsonz .parse.log<rXz.pdf put_object.z.json)basepdf_url artifact_urls marker_urllog_urlz)parse submitted to the GPU worker as job zparse complete: pagesz pages, tablesz tables, total_secondssN) r(r4ryr@PARSE_ARTIFACTSitemsrTrPARSE_DEADLINE_SECONDSrPvalues download_filestr)r%workdirrr>prefixrrr*r]namestemrQrjrs rparsers(0NF8G9$89J ,G B IIK" EL&HWIT2JK.335 5  (<8G9AdV51QR R5 |V@L&: GW F 3F8<= f';QSV`e fF 6::g./x 88L7MYW]WaWabqWrVsstu &&( 8G9AdV5!A3wS\\]^b]cchQiGiCjk) s&!E c [U5up4UUS3nUUS3nUUS3nUUS3n[5n [R"5S- n U RX5[R "SU05R S5S9 S U[S X55[S X65[S X75[S X85S .n [U 5n U"S [U5SU 35 [X7U[X,U 5n [R"U RX6S9SR55Sn[U5[U5:wa$[S[U5S[U5S35eU"S[U5SU R!S5S35 U$)zEmbed the chunks on the worker. The texts go up as one file, the vectors come back as one file, both kept in the document's prefix.z .chunks.jsonz .vectors.jsonz.embed.complete.jsonz .embed.logrtextsre)r7r8rWembedrXr)taskr chunks_url vectors_urlrrz embedding of z+ chunks submitted to the GPU worker as job r6rWvectorszworker returned z vectors for z chunkszembedding complete: z vectors in rr)r(r4ryrrGr~encoder@rTrrEMBED_DEADLINE_SECONDSr`rXr\r{rP)r%rrr>r chunks_key vectors_keyrrr*r]rQrjrrs rrrs(0NF8G9L1JHWI]3K8G9$89J ,G B IIK" EMMdjj'5AQ6R6Y6YZa6bMc|V@ fB|V@L&: GW F-E |#NvhWX f';QSV`e fFjjfFvNSSUVW`aG 7|s5z!-c'l^=U T[\]] s7|nLO9T8UUVWX Nr)returnr)r%rrztuple[str, str])r=rr>rr?rrr)rQDict[str, Any]rr)r>rr?rr]floatrzOptional[bytes])r>rr?rr]rrzOptional[Dict[str, Any]])r>rr?rr]rr Optional[str])rjrrr)r>rrrrrrrrLogrjrr]rrr)r%rrrrrrNone)r%rrz List[str]rrrzList[List[float]]),__doc__ __future__rrGrypathlibrtypingrrrrr urllib.parser r/rLbotocore.configr logurur quber.settingsr r<rrrrrrrrr(r4r@rTr^rcrhrkrrrOrr3rrrs$# 66! "' !  ) wWud{O G   # : I 2! 2! 2!+82!DI2!PS2!]`2!in2!2!jl<6<[[r