j% SrSSKJr SSKrSSKJr SSKJrJrJ r SSK J r SSK J r SSKJr SS KJr SS KJr \"S S 55rS rSrSrSrS\-rSSSjjrSSSjjrSSSjjrSSjrg)uHybrid retrieval over the pgvector playground schema. Two stages. First, two rankings over one document's chunks are fused: pgvector cosine distance (meaning — a paraphrased question still lands) and Postgres full-text search over the same content (letter — a question that names a statement or line item verbatim finds it even when the whole-chunk embedding dilutes the match). Reciprocal rank fusion combines them without tuned weights; a query whose keywords match nothing degrades to the pure vector ranking. Second, a ranker orders the fused list, because rank order alone was measured insufficient: on the evaluation corpus the answering chunk sat as deep as fused rank 63 (a yield buried under same-vocabulary neighbors) or rank 32 (a guidance row losing to on-vocabulary sibling tables), and no scoring change recovered them. Which ranker runs is the playground's `ranker` setting, and both satisfy `ranking.ChunkRanker`. With `haiku`, the default, a selection agent reads a wide, source-diverse window of the fused ranking as one prompt and picks which chunks the question actually needs; the final context is composed from its picks, backfilled from fused order. The agent stage answered all eleven evaluation probes, deterministically, where fused order alone answered eight. Before the agent sees the window, line records are capped at two per parent table — one table's near-identical rows otherwise fill the window and crowd out other sources; each table's whole-table chunk is never capped, so the full table remains available even when its rows are. With `jev`, every page and every chunk of the whole fused list is judged in its own request, so neither the window nor the cap applies: both exist only because an LLM reads the pool as one prompt. The context is the chunks that cleared the ranker's cut, highest score first, then the chunks whose grading calls failed twice, in fused order, up to `k`. Nothing else is backfilled, because a chunk added from fused order is one Jev graded as not bearing on the question or one on a page Jev rejected. ) annotationsN) dataclass)DictListOptional)logger)db) embed_query) ChunkRanker) get_settingscV\rSrSr%S\S'S\S'S\S'S\S'S\S 'S rS \S 'S rg )RetrievedChunk3strchunk_id chunk_typeintpagecontentfloatscoreNz Optional[str]parent_chunk_id)__name__ __module__ __qualname____firstlineno____annotations__r__static_attributes__r8/home/mande/repo/quber/src/quber/playground/retrieval.pyrr3s%MO I L L%)O])r rZc &# [5RRnUS:Xa$SSKJn Un[ X[ U5IShvN nSnO1SSKJn U n[[ X[U5IShvN [5nSnU"X5IShvN n /n [5n U HTnX~nUR U ;a,U R#UR 5 U R%U5 ['U 5U:dMT O U(aXUHRn['U 5U:a OBUR U ;dM&U R#UR 5 U R%U5 MT [)U5VVs0sHunoR US -_M nnn[R*"S U['U 5['U5U Vs/sHnUUR PM sn5 U $GNGNbGNI![a%n [R"SXK5 /SpSn A GNqSn A ff=fs snnfs snf7f) zThe playground's retrieval: the fused list, ordered by the configured ranker. Falls back to fused order if the ranker fails, so a provider outage degrades ranking quality instead of breaking retrieval. jevr)rankNF)selectTz2{} ranker failed ({}); falling back to fused orderz6{} ranker sent {} of {} candidates; fused positions {})r playgroundrankerquber.playground.rankingr( _fused_window FUSED_LIMITquber.playground.selectionr)_cap_line_recordsWINDOW PARENT_CAP Exceptionrwarningsetraddappendlen enumeratedebug)doc_keyqueryktypesnamer(r,poolbackfillr)pickedexcoutseenjcnpositions r!retrieverKTs > $ $ + +D u}1"7;FF5 }WVU'S!SU_`$e** !#CUD  G ::T ! HHQZZ JJqM s8q=  A3x1}zz%$ 1 /8o>oda AE!oH> LL@  C D '*+s!!** s+  JQG "T+ $KTWt$*? ,sAHG (H,G-H= GG G AH//H">H H<-H)H  HHG HG>8H>HHcB[R"[XX#55$N)asynciorunrK)r<r=r>r?s r! retrieve_syncrPs ;;x9 ::r c .# [R"[U5IShvN nU(aSOSnSUSUS3nX@U[[US.nU(aX7S'[ R "5IShvN nURXg5IShvN R5IShvN n SSS5IShvN W V s/sH,n [U S U S U S U S [U S 5U SS9PM. sn $NNNhNTNF!,IShvN (df  N[=fs sn f7f)a The top `window` chunks of the fused vector+keyword ranking. Nothing here holds the event loop. The query embedding is a model forward pass on this host, compute with nothing to await, so it runs in a worker thread; the fused query is awaited on an async connection. Nz!AND c.chunk_type = ANY(%(types)s)a) WITH vec AS ( SELECT c.id, row_number() OVER (ORDER BY c.embedding <=> %(qvec)s::vector) AS rank FROM ade_playground.chunks c JOIN ade_playground.documents d ON d.id = c.document_id WHERE d.doc_key = %(doc_key)s AND c.embedding IS NOT NULL u ORDER BY c.embedding <=> %(qvec)s::vector LIMIT %(pool)s ), -- OR-joined lexemes of the question: AND semantics (websearch_to_tsquery) -- fails whole questions, since one filler word absent from a chunk -- ("find", "tell") kills the match. ts_rank_cd with log-length -- normalization (flag 1) then rewards the chunk covering the most -- question terms, densest first — unnormalized, a boilerplate table -- repeating one query word hundreds of times (a subsidiary list -- repeating the company name) outranks a short exact-coverage line. oq AS ( SELECT to_tsquery('english', COALESCE(NULLIF(array_to_string( tsvector_to_array(to_tsvector('english', %(q)s)), ' | '), ''), 'zzznomatchzzz')) AS q ), kw AS ( SELECT c.id, row_number() OVER ( ORDER BY ts_rank_cd(c.content_tsv, oq.q, 1) DESC ) AS rank FROM ade_playground.chunks c JOIN ade_playground.documents d ON d.id = c.document_id, oq WHERE d.doc_key = %(doc_key)s AND c.embedding IS NOT NULL a3 AND c.content_tsv @@ oq.q LIMIT %(pool)s ), fused AS ( SELECT COALESCE(vec.id, kw.id) AS id, COALESCE(1.0 / (%(rrf)s + vec.rank), 0) + COALESCE(1.0 / (%(rrf)s + kw.rank), 0) AS score FROM vec FULL OUTER JOIN kw ON kw.id = vec.id ) SELECT c.chunk_id, c.chunk_type, c.page, c.content, fused.score, c.parent_chunk_id FROM fused JOIN ade_playground.chunks c ON c.id = fused.id ORDER BY fused.score DESC LIMIT %(cand)s )qvecr<qrArrfcandr?rr*r%)rrrrrr) rN to_threadr POOLRRF_Kr connect_asyncexecutefetchallrr) r<r=windowr?rS type_filtersqlparamsconnrowsrs r!r.r.s%"";6 6D9>5BKG HSmTG.HSmT9* CVUDQV`f gF w!!T LL55??AA"! A qTt1aD!+aD    g 7b"5A"!!! sDC,A D.C./D2C6C0C6C2C6" D-C4.D63D)D.D0C62C64D6D <C? = D  Dc0n/nUHinURS:XaEUR(a4URURS5nXQ:aMGUS-X$R'URU5 Mk U$)uKeep at most `cap` line records per parent table, in fused order. Everything that is not a line record — prose, figures, and each table's whole-table chunk — passes through untouched, so a capped table's full content is still in the pool. line_itemrr*)rrgetr8)candscaprFrErHrIs r!r1r1snD "C  <<; &1+<+<**A.Ax&'!eD"" # 1  Jr ) N) r<rr=rr>rr?Optional[List[str]]returnList[RetrievedChunk]rM) r<rr=rr`rr?rmrnro)rjrorkrrnro)__doc__ __future__rrN dataclassesrtypingrrrlogururquber.playgroundr quber.playground.embeddingr r-r quber.settingsr rr\r[r2r3r/rKrPr.r1rr r!rxs"H#!''20' ** *     $h IM6 66!$62E66tIM; ;;!$;2E;;IMF FF&)F2EFFRr