j IVSrSSKJr SSKrSSKrSSKJr SSKJr SSK J r J r J r J r JrJrJrJrJr SSKrSSKJr SSKJrJr SS KJrJrJrJrJrJrJ r J!r! SS K"J#r# SS K$J%r% SS K&J'r'J(r( SS K)J*r* \ (aSSK+J,r, "SS\5r-Sr.Sr/Sr0Sr1Sr2Sr3\"\.S9r4\"\/\0\1\2\3/S9r5Sr6"SS\5r7"SS\5r8"SS\5r9\"S S!9S/S"j5r:S0S#jr;\"S$5r<"S%S&5r=S1S'jr>S2S(jr?S3S)jr@S4S*jrAS5S+jrBS6S,jrCS7S-jrDS7S.jrEg)8aJev ranking of retrieved chunks for the playground's answer path. Ownership. This module decides which of a question's retrieved chunks the answer model reads when the playground's ranker setting is `jev`. `selection` holds the Haiku equivalent, and `retrieval.retrieve` chooses between them and composes the final context. Both satisfy `ChunkRanker`: the question and the candidates in, indices into the candidates out, most relevant first. How it ranks. Jev, TypeSafe's scoring model, does not read the candidates as one prompt. Every page and every chunk is judged alone, in its own request, so the ranker can afford the whole fused list rather than a window of it. Two passes. Each distinct page in the candidate list is asked whether it has material relevant to the question, and a page answered no is dropped with every candidate on it. Each candidate on a page that passed is then graded on a four-level rubric, and `Ranking.order` keeps the ones at or above the configured cut, highest first. Nothing is added back from fused order when fewer clear the cut: anything added that way is a chunk Jev graded as unrelated or same-topic, or one on a page Jev rejected. What leaves the host. The question, each page's text and each chunk's text are sent to TypeSafe's API, one text per request. TypeSafe states it does not train on requests; zero data retention is an enterprise option not yet taken. Failures. The SDK retries 429, 5xx, connection and timeout errors itself. After each pass, the calls that still failed are run once more. A page that fails twice counts as relevant, so its chunks are still graded. A chunk that fails twice goes after the graded ranking in fused order. Every failed page or chunk is named in the log and the trace, because it may be the one that holds the answer. `rank` raises when every call in a pass fails, and on any error that is not a `TypeSafeError`, such as a missing TYPESAFE_API_KEY or a failed page-text query. `retrieve` then falls back to fused order. ) annotationsN)Counter) lru_cache) TYPE_CHECKINGAny AwaitableCallableDictListOptionalProtocolTypeVar)logger) BaseModelField)AsyncTypeSafeClientNoul NoulAnswer RetryPolicyScore ScoreAnswerTypeSafeAPIError TypeSafeError)usage_metadata_from)db) run_metadatatracer) get_settings)RetrievedChunkc"\rSrSrSrSSjrSrg) ChunkRankerKzVThe question and a candidate list in; indices into that list out, most relevant first.c # g7fN)selfquestioncandss 6/home/mande/repo/quber/src/quber/playground/ranking.py__call__ChunkRanker.__call__NsY\sr%Nr'strr(List['RetrievedChunk']return List[int])__name__ __module__ __qualname____firstlineno____doc__r*__static_attributes__r%r)r!r!Ks`\r7r!zDoes this page have material relevant to the query? Yes when the page discusses the metric, line item, or statement the query asks about. No when the page is about other subjects and nothing on it bears on the query.z(How does this chunk relate to the query?zIThe chunk is about a different subject; nothing in it bears on the query.zpThe chunk is on the same topic as the query but gives neither the asked-for item nor anything that qualifies it.z~The chunk supports the asked-for item: a definition, a footnote, a component, a reconciliation, or a comparison period for it.zXThe chunk states the asked-for item directly, for the entity and period the query names.) instructions)r8criteria?cJ\rSrSr%SrS\S'S\S'SrS\S 'SrS \S 'S rg) PageScore~z7One page's verdict. `page` is the stored 0-based index.intpageboolrelevantNOptional[float] confidence Optional[str]errorr%) r1r2r3r4r5__annotations__rCrEr6r%r7r)r<r<~s#A IN"&J&E=r7r<cb\rSrSr%SrS\S'S\S'S\S'SrS \S 'SrS \S 'SrS \S'Sr g) ChunkScorezEOne candidate's grade. `index` is its position in the candidate list.r>indexr-chunk_idr?Nz Optional[int]graderBscorerDrEr%) r1r2r3r4r5rFrLrMrEr6r%r7r)rHrHs4O JM IE="E?!E=r7rHc\rSrSr%SrS\S'S\S'S\S'S rS \S 'S rS\S 'SrS\S'\ "\ S9r S\S'SSjr SSjr SSjrSrg )Rankingz1Everything both passes returned for one question.zList[PageScore]pageszList[ChunkScore]scoresfloat grade_cutNrDmodelsecondsrr> input_tokens)default_factoryzDict[str, int]failurescURVs/sH/oRcMURUR:dM-UPM1 nnURSSS9 URVs/sHoRcMUPM nnUVs/sHoR PM snUVs/sHoR PM sn-$s snfs snfs snfs snf)zsCandidate indices at or above the cut, highest score first, then the chunks whose calls failed, in candidate order.c,UR=(d S$NrVrM)cs r)Ranking.order..s!''.S.r7Tkeyreverse)rRrMrTsortrErJ)r&r_gradedfaileds r)order Ranking.orders"[[^[GG!AGGt~~D]![^ 0$ ?![[@[GG![@!'(A(V+DVGGV+DDD_@(+Ds'B;B;B;&C:CC!C cB^[U4SjTR55$)Nc3|># UH1oRcMURTR:dM-Sv M3 g7f)N)rMrT).0r_r&s r) "Ranking.cleared..s+_kWW1QWWPTP^P^E^11ks << <)sumrRr&s`r)clearedRanking.cleareds_dkk___r7c~URVs/sH!oR(dMURPM# sn$s snfr$)rQrAr?)r&ps r) kept_pagesRanking.kept_pagess( $ 9 1jj 999s::r%)r/r0)r/r>)r1r2r3r4r5rFrUrWrXrdictrZrhrrrvr6r%r7r)rOrOsW;  E=GUL# %T:Hn:E`:r7rOrl)maxsizec[5RnUR(d [S5e[R "UR [R"SSS9S9n[UR[URS9US9$)asThe one SDK client both passes share, so every call reuses one connection pool. The pool is unbounded: a batch of questions, each with `concurrency` calls in flight, would otherwise queue on httpx's default of 100 connections and reopen the ones past its keep-alive cap of 20 on every call. The per-question semaphore and the batch's question cap are what bound the load.z5TYPESAFE_API_KEY is not set; the jev ranker needs it.N)max_connectionsmax_keepalive_connections)timeoutlimits) max_retries)api_keyretry http_client) rtypesafer RuntimeErrorhttpx2 AsyncClienttimeout_secondsLimitsrrr)tshttps r)clientrso  B ::RSS   ""6==im+n D  +".."IW[ r7c# [UVs1sHoRiM sn5nUVs/sHoRPM nnSnUVs0sHoU/_M nn[R"5IShvN nUR XCUS.5IShvN R 5IShvN HupXhRU 5 M SSS5IShvN UR5VV s0sHupZUSRU 5_M sn n$s snfs snfs snfNNNxNN!,IShvN (df  Nc=fs sn nf7f)aThe text of each distinct candidate page: its chunks that are not line records, joined in stored order. Candidates carry no document key, so the pages are read from the document that holds the most of the candidates' chunk ids. Those ids are docling refs and positional names such as `#/texts/12`, `#/tables/3` and `t3-line-5`, unique only within a document. Another document that holds every candidate id ties with the right one, and the query does not break the tie, so the pages can come from the wrong document. a WITH doc AS ( SELECT document_id FROM ade_playground.chunks WHERE chunk_id = ANY(%(ids)s) GROUP BY document_id ORDER BY count(*) DESC LIMIT 1 ) SELECT k.page, k.content FROM ade_playground.chunks k JOIN doc ON doc.document_id = k.document_id WHERE k.page = ANY(%(pages)s) AND k.chunk_type <> 'line_item' ORDER BY k.page, k.id N)idsrQz ) sortedr?rKr connect_asyncexecutefetchallappenditemsjoin) r(r_rQrsqlrutextsconnr?contentpartss r) page_textsrs E*EqFFE* +E$ %u!::uC % C38"8%Qb5%E"8!!T*.,,sRW# TIShvN T"U5IShvN sSSS5IShvN $N%NN !,IShvN (df  g=f7fr$r%)itemcallsems r)guarded_twice..guardeds,3d#33#333sEA0A 626 A4A6AA ? A  Ac34># UH nT"U5v M g7fr$r%)rmirs r)rn_twice.. s)De'!**esNc3:># UHnT"TU5v M g7fr$r%)rmnrrs r)rnr s&H%QwuQx'8'8%sT)strict)rrr/r)asyncio Semaphorelistgather enumeratezip) rrrg concurrencyresultsrrragainrrs `` @@r)_twicers   K (C$$)De)DEEFG$W- ;-41Q-E ; nn&H%&HIIT2DAAJ3 N F ;Is6AC C C"C 9C ?+C*C+C Cc lSRS[[[[[ 4555$)N c34# UHupUSU3v M g7f)rNr%)rmrds r)rn_levels..s  l1#Rs ls)rrLEVEL_UNRELATEDLEVEL_SAME_TOPICLEVEL_SUPPORTS LEVEL_STATESr%r7r)_levelsrs0 99 )?# TRUS5nU(d [USS9$[5RSTSUS-SU3S[0TR S 9IShvN nTRU5 URSn[U[5(de[UUR[:[[UR[- 5S -S 5S 9$N~![ a$n[USTRU5S 9sSnA$SnAff=f7f)NT)r?rAQUERY: z PAGE rlz: rArU)r?rArEr)r?rArC)getr<r system_onePAGErUrrgranswersrrnoulPAGE_THRESHOLDrabs) r?textrranswerr'rrrs r)judgejudge_pages..judge/syyr"$6 6 P#X00(9TAXJc$@:tBT\^\d\d1H  X!!*-&*----[[N2S~!=>BAF   P$U\\#=NO O PsG$D8C! C!C!%A:DC!! D+D DD DD page_rankrUextra_metadatacURSL$r$rErus r)r`judge_pages..Gs QWWD=Pr7c3<# UHoRSLv M g7fr$rrmrus r)rnjudge_pages..I>v!''-vzevery page call failed: c<UR(+UR4$r$)rAr?rs r)r`rKs1::~qvv.Fr7)rc assistantc3J# UHoR(dMSv M g7f)rlN)rArs r)rnrQs41QQs# #)rrQkept)r?r>r/r<)rrrr PAGE_QUESTIONrrrrllm_runrUrrrrr?allrrxrZrr-rA model_dumprpoutputs) r'r(ruinputsrrrunrQjudgedorderedrrrs ` @@@r) judge_pagesr $s  B HEU# #E&=AF_gChi!'/Aa%/F   ,!Gx V288T`TbccgjVE]E3PRTR`R`aa"8Ueii&7!U8 c>v>>>!9$u~~:N9OPQ Q$FG"  *sPU;dPU1YcYcJAFFQJPU;d7efgCJK7a9alln9faffqj97K444   dc %<U $04da8H$? I IH&H&$I&I,H/ - I9 Ic B^^^# [5Rm[5m[U5VVs/sHup4URU;dMX44PM nnnS[ S[ 53S.STS./UVVs/sHupdURPM snnS.nSUUU4Sjjn[R"5n [5RSUTR[5S9IS hvN n [XXS TR5IS hvN n U (a8[!S U 55(a![#S [%TR&535e[)S U 5SSS9n U V s/sHoR*cMU PM nn U V s/sH)oR,=(d STR.:dM'U PM+ nn [1TU S[3UV s/sHoRPM sn 5S./X-V s/sH&n 0U R55ESU RS-0EPM( sn TR.[7U5[7U 5[7U5- S9U lS S S 5IS hvN U T4$s snnfs snnfGNGNgs sn fs sn fs sn fs sn fN.!,IS hvN (df  W T4$=f7f)zPass 2: one rubric call per candidate on a kept page. One traced run named `chunk_rank`, with every score, best first, failed chunks last.rrrr)r chunk_idsc ># UupSTSURS-SURSUR3n[5R US[ 0T R S9IShvN nTRU5 URSn[U[5(de[UURUR[[!UR"S -5[%[ R&5S- 5UR"S 9$N![a8n[XRURTRU5S9sSnA$SnAff=f7f) Nrz CHUNK (page rlz, z): rLr)rJrKr?rEr:)rJrKr?rLrM)r? chunk_typerrrCHUNKrUrrHrKrgrrrrminr>rMlenr9) rrr_promptrrrr'rrs r)rLgrade_chunks..gradefs#8*$4QVVaZL1<<.PTUVU^U^T_` b#X00'59IQSQYQY0ZZH  X!!'*&+....ZZc&,,,-s5>>/BQ/FG,,   [ bA u||\_O`a a bsG4E ,D$D%D)BE D E -E>E ?E E  E  chunk_rankrNcURSL$r$rss r)r`grade_chunks..zs aggT6Ir7c3<# UHoRSLv M g7fr$rrmrs r)rngrade_chunks..{rrzevery chunk call failed: c3B# UHoRbMUv M g7fr$rrs r)rnr}s>FqggFs c,UR=(d S$r]r^rs r)r`r}saggnY\nr7TrbrVrr?rl)rrRrTrr below_cut)rztuple[int, 'RetrievedChunk']r/rH)rrrrr?CHUNK_QUESTIONrrKrrrrrUrrrrrrxrZrrErMrTrr-rrr)r'r(rrr_r_r rLrr rRrfrrgrrrrs` @@r) grade_chunksr$Vs=  B HE )% 0 C 0AFFdNVaV 0E C^,>ZZ c>v>>>!:4;O:PQR R>F>D\fjk#;Vww!V;$IfCBLL(H1fI"  *sPW;XPW1JJPW;X7YZ[DJOTOq:q||~:vqvvz:OTllL&kCL0  ed 5=Y D 4,eZ<IBoth passes over the candidates, with everything Jev returned.N)rQrRrTrUrWrXrZ)rrrrr rAr?r$rOrTrUrXrxrZ) r'r(rrrQ page_tallyrurrR chunk_tallys r)rMrMs  B!G)(::E! 0EqZZFAFFED 0 ,Xd CCF ,,3:#3#3!!#g-,,{/G/GGj))K,@,@@A ; 0Cs48DD DD D -D>D?B D Dc # [X5IShvN nUR5n[R"S[ UR 5[ UR 55[ UR5[ U5UR5URURURUR=(d S5 UR Vs/sHoDR(dMUPM nnURVs/sHofR(dMUPM nnU(dU(a{[R"S[ U5[ U5UVs/sHoDRS-UR4PM snUVs/sHofR UR4PM sn5 URR#S5(a$[R"SURS5 U$GNs snfs snfs snfs snf7f)zDIndices into `cands` in the order the answer model should read them.Nzjev ranker: {} pages judged, {} kept; {} of {} candidates graded, {} cleared the cut of {}; {:.1f}s, {} input tokens, failures {}nonezNjev ranker: {} page calls and {} chunk calls failed twice; pages {}; chunks {}rl429zdjev ranker: {} calls were throttled (429) past the SDK's retries; TypeSafe's limits may have changed)rMrhrdebugrrQrvrRrrrTrWrXrZrEwarningr?rKr)r'r(rankingrhru failed_pagesr failed_chunkss r)rankr0s}(**G MMOE LL 0 GMM G   ! GNN E "F  '}}8}!A}L8 ':1''QM:} \    ,8 9Lqffqj!'' "L 9,9 :Mqjj!'' "M :  E"" r   U #  L=+9: : :sSH G5CH G80G86H G=G=$9H $HH !H(AH 8H c@[R"[X55$r$)rr r0)r'r(s r) rank_syncr2s ;;tH, --r7)r/r)r(r.r/zDict[int, str]) rzList[T]rzCallable[[T], Awaitable[Any]]rgzCallable[[Any], bool]rr>r/z List[Any])r/r-)rrrrSrrr/zDict[str, Any])r'r-r(r.r/ztuple[List[PageScore], _Tally])r'r-r(r.rzset[int]r/ztuple[List[ChunkScore], _Tally])r'r-r(r.r/rOr,)Fr5 __future__rrr collectionsr functoolsrtypingrrrr r r r r rrlogururpydanticrr typesafe_sdkrrrrrrrrquber.agents.langsmith_tracerrquber.playgroundrquber.playground.tracingrrquber.settingsrquber.playground.retrievalrr!rr"rrrrrrrr<rHrOrrrrrrrr r$rMr0r2r%r7r)r?sB#    %   >9'9](]U<]& 7j  ' ? CL* 7AVeh( /d33038@3$3l$ F.r7