pyiSSKJr SSKJr SSKJr SSKJr SSKrSSK J r SSK J r SS K Jr S S KJr S S KJr S S KJrJrJr S SK JrJrJrJrJr \"SS55r"SS5r"SS5r"SS5rg)) nullcontext) dataclass)partial)AnyN)PretrainedConfig)get_available_devices)traced)PagedAttentionCache)%ContinuousBatchingLogitsProcessorList) TMP_TOKEN_IDFutureRequestStatelogger)CudaGraphBufferaligned_divideattn_mask_is_neededbuild_attention_mask pad_to_pow2c\rSrSr%Sr\R \S'\R \\ \R 4-S-\S'\R \S'\R \S'\R \\ \R 4-\S'\ \S '\ \\ \ 4-\S '\ \R \S '\ \R \S '\R \S '\ \S'\R S-\S'\R \S'Sr \\S'S\\ \44SjrSrg)PagedAttentionArgsaDataclass containing the keyword arguments for a forward pass using paged attention. Attributes: input_ids: Input token IDs tensor of shape `(1, total_query_tokens)`. attention_mask: Attention mask tensor or dictionary mapping layer types to masks. Can be `None` if the attention implementation doesn't require explicit masks. position_ids: Position IDs tensor of shape `(1, total_query_tokens)`. cu_seq_lens_q: Cumulative sequence lengths for queries, used for variable-length batching. cu_seq_lens_k: Cumulative sequence lengths for keys/values. Can be a tensor or dictionary mapping layer types (e.g., "full_attention", "sliding_attention") to tensors for hybrid models. max_seqlen_q: Maximum query sequence length in the batch. max_seqlen_k: Maximum key/value sequence length. Can be an int or dictionary for hybrid models. write_index: List of tensors indicating where to write new KV states in the cache, one per attention group. read_index: List of tensors indicating which cache positions to read from, one per attention group. logits_indices: Tensor indicating which positions in the output should be used for next-token prediction. cache: The [`PagedAttentionCache`] instance managing the KV cache. block_table: Block table for paged KV cache. If provided, uses `flash_attn_with_kvcache` for fused attention + cache update. More information in src/transformers/integrations/flash_paged.py logits_processor_args: List of tensors containing the arguments for the logits processors, one per request. use_cache: Whether to use caching (always `False` in continuous batching as the cache is managed externally). input_idsNattention_mask position_ids cu_seq_lens_q cu_seq_lens_k max_seqlen_q max_seqlen_k write_index read_indexlogits_indicescache block_tablelogits_processor_argsF use_cachereturnc<URURURURURUR UR URURURURURURURS.$)Nrrrrrrrr r!r"r#r$r%r&r)selfs v/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/generation/continuous_batching/input_outputs.pyasdictPagedAttentionArgs.asdictFs"11 --!//!// -- --++//"11ZZ++%)%?%?  )__name__ __module__ __qualname____firstlineno____doc__torchTensor__annotations__dictstrintlistr r&boolrr-__static_attributes__r0r/r,rrs,||LL4U\\(9#::TAA,,<<<<$sELL'8"999S#X&&ell##U\\""LL $$ <<'It S#X r/rcT\rSrSr%SrSr\\S'S0S\S\ S\ RS\ RS \S \ S \S \ S S4Sjjr\"SS9S \S S4Sj5rS0SSS\ R$R&S\ S S4Sjjr\\ R*"5S0S\ S S4Sjj55rS1SjrS \\ R2\\\ R2444SjrS\ R2S\ R2S\ R2S S4SjrS1SjrS \\\ \\\\!S-44S jr"\S!\\ S"\S#\ S$\S%\S S4 S&j5r#S0S'\ S \\\$44S(jjr%S \\ R2\ R2\ R244S)jr&S \\S*44S+jr'S \ R$RPS-4S,jr)S-\ R$RPS S4S.jr*S/r+g)2ContinuousBatchingIOsYzA class to hold inputs and outputs for a continuous batching forward pass, using static tensors as storage. The class is meant to be self-contained, so once a set of inputs have been created, the class can be used to update the batch alone.  static_inputsr#configdevice model_dtype max_graphsreturn_logprobslogit_processoruse_cuda_graph_varlenr'Nc XlX0lX lX@lXl[ USS5cSO UR UlX`lSUlSUl SUl [UR5V s/sHn SPM sn Ul [UR5V s/sHn SPM sn UlSUl/Ul0Ul[%U5UlUR(UlUR-US9 UR/SS9 UR0S :Xa-[2R4R7URS 9UlgSUlgs sn fs sn f) aInitialize the continuous batching I/O manager. Args: - cache: The [`PagedAttentionCache`] instance managing the KV cache. Meant to be unique. - config: The model's pretrained configuration. - device: The device to allocate tensors on. If the device is CPU, then the memory is pinned. - model_dtype: The data type for model computations. - max_graphs: Maximum number of CUDA graphs to cache. Uses LRU eviction when full. - return_logprobs: Whether to return log probabilities along with the token IDs. - logit_processor: The [`ContinuousBatchingLogitsProcessorList`] object used to process the logits. - use_cuda_graph_varlen: Whether CUDA graphs are enabled for the varlen (prefill) path. sliding_windowNr rF)rIT full_resetcudarE)r#rErDrFrJgetattrrLrH num_q_tokens max_kv_readtrue_batch_sizerange num_groupstrue_read_sizestrue_write_sizesuse_block_tablerequests_in_batchreq_id_to_new_token_positionrgraphs trash_index _trash_index_setup_static_tensors_reset_static_tensorstyper6rOStreamcompute_stream r+r#rDrErFrGrHrIrJ_s r,__init__ContinuousBatchingIOs.__init__as4,   &%:"#*63CT#J#RaX^XmXm. +01A1A+BC+Ba+BC,1%2B2B,C D,Cq,C D$;=<>)'6z'B !-- ""?"C ""d"3GM{{V\G\ejj//t{{/Cbf D Ds 2 E  ET) standalonecURRnURRnURRURR-nUR R S:H=(a [[55S:nURUR-n[US-SS5n[R"Xg4[RUR US9Ul[R"URS4[RUR S9UlUR#UR 5 URSSU24UlURSSU24UlURSSUS-24UlURS SU24UlURS SUS-24nURS SUS-24n URS SU24Ul0UlURR0(aXR.S 'URR2(aXR.S'UR4(aSOSn [R"XS-4[RUR US9UlUR6R95 SUlSUl[>RAUR.RC5S5Ul"[GURH5(ag0Ul%UR.RC5HAn [R"SSX4U-4URLUR US9URJU 'MC OSUl%URRNS:aUOSn [R"XURRN4[RUR US9Ul([R"X#4[RRUR US9Ul*[R"X$U-4[RRUR US9Ul+g)aAllocates static tensors for generation inputs and outputs. This is called only once at init time, to avoid repeated allocations and enable CUDA graphs. All tensors are allocated with maximum possible sizes. The allocated tensors are: - `_bulk_input_tensor`: Storage for all the small inputs: `input_ids`, `position_ids`, `cumulative_seqlens_q`, `logits_indices`, `cumulative_seqlens_k`, `carry_over_ids`. - `attention_mask`: Optional attention masks (only for eager/SDPA implementations) - `write_index` and `read_index` storage: Cache indexing tensors for each attention group - `output_ids`: Storage for generated token IDs and maybe log probabilities if return_logprobs is True cpur )dtyperE pin_memoryrlrErNrfull_attentionsliding_attention)sizerlrErm),r#rVmax_batch_tokens num_blocks block_sizerEralenr rCtensors_requiredrr6emptyint32_bulk_input_tensorlogits_processors_defaults fill_defaultsrrcumulative_seqlens_qr"carry_over_idscumulative_seqlens_knum_full_attention_groupsnum_sliding_attention_groupsrH output_idszero_total_seqlen_qrr9fromkeyskeysrrrDrrFmax_blocks_per_requestr$int64write_index_storageread_index_storage) r+rIrVrv num_pagesrm bulk_lines bulk_columns#full_attention_cumulative_seqlens_k&sliding_attention_cumulative_seqlens_knum_output_rows layer_typens r,r_+ContinuousBatchingIOs._setup_static_tensorssZZ** ::66JJ))DJJ,A,AA [[%%.S37L7N3ORS3S ''/*J*JJ %&6&:ArB "'++  &ekk$++Zd# +0++  - -q 1T[[+ ' %%d&E&EF004E5E4E1EF 33A7H8H7H4HI$($;$;A?UAQTUAU?U@! :: / /:] % %&6 7 :: 2 2=c % %&9 : $33!++ 2 35;;t{{gq   MM$*C*C*H*H*JAN t{{ + +"$D "77<<> 27++Q 0>N2NO**;;) 3##J/?#'D **;;a?JQ ;; $**"C"C D++;;!  $);;  *%++dkk^h$  #(++ %55 6ekkRVR]R]jt# r/otherstream non_blockingc.URUlURUlURUlURSSUlURSSUlUR UlUR UlURUl[URR55Ul Ub[RRU5O [5nU URR!URUS9 UR (a%UR"R!UR"US9 OXUR$R!UR$US9 URS:a$UR&R!UR&US9 UR(bXUR(bKUR(R+5H-nUR(UR!UR(UUS9 M/ SSS5 g!,(df  g=f)Nrr)rRrSrTrWrXrYrrr9ritemsr6rOrrr}copy_r$rrrr)r+rrr maybe_streamrs r,_transfer_inputs&ContinuousBatchingIOs._transfer_inputss".. ,, $ 4 4 $ 4 4Q 7!%!6!6q!9 $ 4 4#22!..!$"3"3"9"9";<4:4Fuzz((0KM   $ $ * *4+B+BQ] * ^##!!''(8(8|'T))//0H0HWc/d##a',,2243J3JYe2f"".53G3G3S"&"5"5":":"N>N0NOUUKK 0 01554    QY ' - -b 1  $ $QY / 5 5d6G6G H  # #A'7'7$7 8 > >t?P?P Q  ! !   QY ' - -b 1  $ $QY / 5 5d6G6G H  # #A'7'7$7 8 > >t?P?P Qr/cURSS9 /Ul0UlURbURR 5 gg)z4Reset all relevant states for a new generation loop.TrMN)r`rZr[rc synchronizer*s r,resetContinuousBatchingIOs.reset0sF ""d"3!#,.)    *    + + - +r/c2URUR4$)z:Get the cumulative sequence lengths for the current batch.)rrr*s r,get_cumulative_seqlens,ContinuousBatchingIOs.get_cumulative_seqlens9s(($*C*CCCr/rrprev_output_idscgNr0)r+rrrs r,carry_over_tokens'ContinuousBatchingIOs.carry_over_tokens=s r/cTURbURR5 ggr)rcrr*s r,retrieve_device_outputs-ContinuousBatchingIOs.retrieve_device_outputsBs%    *    + + - +r/cPURnURSS[UR524R5nUR(aPURSS[UR524R [ RS9R5nOSnXU4$)Nrr rl)rZrrytolistrHviewr6float32)r+rZ new_tokenslogprobss r,prepare_batch_update*ContinuousBatchingIOs.prepare_batch_updateFs 22__Q(E#d.D.D*E(E%EFMMO   q*GC0F0F,G*G'GHMMTYTaTaMbiikHH h66r/rZlogits_processorsuse_decode_fast_pathrRrSc  U(d [S5eU=(a URR5S:UlX@lUR(aSOUUl[ U5UlUR5 [URR5Vs/sHnSPM snUl [URR5Vs/sHnSPM snUl /Ul0Ul/n/nS/n /n UR R#5V s0sHoS/_M n n [URR5Vs/sHn/PM n nUR S:XaSO/[URR5Vs/sHn/PM snn[%U5GHunnUR&nUR(nUR*nURR-UU5nU=R(U- slUR/UR05 UR/[UUU-55 U R3U SU-5 [5UR6U5UlUR95HGun nXR3XSU-5 [5UR:U U5UR:U 'MI UR(a:URR=UR>UUURSS2U45 O(URRAUR>UUX5 URB(a?U R3U SS- 5 [D/UlU SURUR>'URR3U5 GM URGUURHURJSS9 [M[NRP[NRRURTS9nU"U5URVS[ U5&U"U5URXS[ U5&U"U 5URZS[ U 5&U"U 5UR\S[ U 5&U SUl/U R95Haun nU"U5UR U S[ U5&UR`cM6[cUR`U U UU S:Xa URdOSS 9 Mc UR(d[M[NRP[NRfURTS9n[%U 5H@unnU"U5URhUS[ U524'[ U5URU'MB UbP[%U5H@unnU"U5URjUS[ U524'[ U5URU'MB gggs snfs snfs sn fs snfs snf) atPrepare tensors and metadata for the next model forward pass, using the given requests as data. This method: 1. Resets the static tensors from the previous batch 2. Iterates through requests to accumulate input_ids, position_ids, and sequence lengths 3. Extends read/write indices for cache management 4. Builds attention masks if needed (for eager/SDPA implementations) 5. Converts accumulated lists to tensors and copies them to static storage This method also modifies the `position_offset` attribute of each request to track progress and adds a temporary token at the end of the requests for which there will a new token. zNo requests in batchrNrr )rZ arg_storagernrt)rrrrL)6 ValueErrorr$numelrYrRrSryrTr`rUr#rVrWrXrZr[rr enumeratestateposition_offset query_length get_seqlens_kextendtokens_to_processappendmaxrrrfill_block_table request_idextend_read_and_write_indices has_new_tokenrprepare_tensor_argsr}rCrr6tensorr|rErrrr"rrrrLrrr)r+rZrrrRrSrerrrr"rrr r!i future_stater past_lengthr seqlens_klayer_type_seqlen_k to_tensorlayer_type_seqlens_kto_index_tensorgroup_write_indicesgroup_read_indicess r,prepare_batch_tensors+ContinuousBatchingIOs.prepare_batch_tensorsQs}*!34 4 4T8H8H8N8N8PST8T( $ 4 41+"#45 ""$,11F1F+GH+Ga+GH,1$**2G2G,H I,Hq,H I!#,.)  !sBFB[B[B`B`BbcBbJQCBbc#()>)>#?@#?ar#? @!--2TU4::K`K`Ea8bEaEa8b  )):;OA| &&E//K'44L 00lKI  ! !\ 1 !   U44 5   k;3M N O ' '(&BC+7.'FTUWFX11%2B2BC  " " ) ), 7G 5>~5N1c.122261E0J0J0L ,J,QZ[oQpD % %j 12MC8L4M N"".$#'#6#6z#B)=)=:DH[:[4#6#6ab 1M##%ell%++dkkZO*3K*@&&JYZmJn((,Fc2E.F,F)FG+./B+C%%a(+A%-6z-B)A)L[\nLoD++A/H5G1H/H,HI.12D.ED((+.C& $e I I d@8bs W  W% W* W/ W4 use_paddingcnURnURUR-nU(a URO URn[URSUR S5UR SUR S5URSUS-URURSUURURS2SU2400URcSO0//URUR(aURSS2SU24OSSS9nU(aUR UR"URS-S&UR(dtUR$(acX lUR&R)5Vs0sH2nU[+UR&UURR,S5_M4 snUlUR(aSO URUl[/URR05HnU(aUOUR2UnUR4R7UR8USU245 URS:XaSn OU(aUOUR:Un UR<R7UR>USU 245 M UR@RC5HupjU SUS-URDU'U(a+XRURDUURS-S&UR(aSOUR&UUR&U'URcMU(aUOXn URUSSU2SU 24URU'M [GUR@R)55S:XasURDRI5SUl"UR&RI5SUlURb"URRI5SUl URK5$s snf)zGet model keyword arguments for the current batch, eventually padding the query dimension and KV dimensions if use_padding is True. The padding is only useful if we want static shapes, like when using cuda graphs.Nrr F)rrrrr"r%rrrr!r r#r$r&i.)&rRrSrTrr unsqueezerrrr"r}rCrr#rYr$rrrJrrrrrUrVrXr rrrWr!rrrrrypopitemr-) r+rq_sizekv_size batch_sizekwargsrrwrite_index_sizeread_index_sizerk_lens r,get_model_kwargs&ContinuousBatchingIOs.get_model_kwargss""""T%6%66*5T&&4;O;O $nnWf-77:**7F3==a@334Dj1nE**..w7"&"9"9$:L:L:NPWQWPW:W"X#'#6#6#>4B**<@%&>  D,=,=j,I4::K_K_ae ff&>%!$(#7#7aT=N=Ntzz,,-A)4v$:O:OPQ:R     % %d&>&>qBSCSBS?S&T U1$"#-8'd>R>RST>U    $ $T%<%%>%D%D%F !J/89I:>/JF  ,OXYmYmOn$$Z01E1E1I1KL373G3GaTM^M^_iMjF   +"".#.I4I484G4G 4STWY`Z`Y`bhchbhTh4i%%j1&G t((--/ 0A 5#)#7#7#?#?#A!#DF "("5"5"="="?"BF "".(.(=(=(E(E(G(J%}}K%s9P2cHURURUR4$)aReturns the tensors used inside the generation step that are not inputs to the model forward pass. In synchronous batching, there is no carry over, so the only tensor that will be used is output_ids, but we still return 3 tensors to have the same interface as when using async batching.)rrr*s r, get_cb_kwargs#ContinuousBatchingIOs.get_cb_kwargss""DOOT__DDr/.cUR(a UR4$URUR/URR 5Q7$r)rYrRrSrvaluesr*s r,_get_graph_key$ContinuousBatchingIOs._get_graph_keysE   %%' '!!4#3#3Qd6G6G6N6N6PQQr/cUR5nURRU5nUc4URR5 [R "SU<35 U$)NzCreating graph for key = )rr\ get_graphplan_for_new_graphrinfo)r+keygraphs r,rContinuousBatchingIOs.get_graph#sR!!# %%c* = KK * * , KK4cX6 7 r/rcUR5nURRX!5 [R"SU<35 g)NzSetting graph for key = )rr\ set_graphrr)r+rrs r,rContinuousBatchingIOs.set_graph,s6!!# c) /12r/)"r}r^rr$r#rrcrDrrrEr\rr"r~rSrrrFrRrrrr[rZrHrLrrTrWrXrYrJrFr'N),r1r2r3r4r5rCr;r8r rr6rErlr=r rfr r_rOrbrno_gradr`rtupler7r9r:rrrr<rfloatrrrrrr CUDAGraphrrr>r0r/r,r@r@Ys M3',,g",g!,g ,g [[ ,g  ,g,g?,g $,g ,g\ tT 5ZT _cT T p_dw,w6;jj6G6GwW[w w>  ]]_&R&R&R &RP.DellDellARr0r/r,rr2s',#R"#R!#R #R [[ #R  #R#R?#R $#R #RJ$X%***;*;XXX5::+<+│1->│ ¦ │2->│ ¦ │3->│ ¦ ¦ │ └───┼───┴───────────┬─────────────┴─┬─┼───────────┴───┼───────────────┐ ¦ GPU │ │ COMPUTE 0 │ COMPUTE 1 │█│ COMPUTE 2 │ COMPUTE 3 │ ¦ │ └───────────────┼───┬───────────┼─┴─┬─────────────┼───┬───────────┼───┤ D2H │ │0<-│ │1<-│ │2<-│ │3<-│ │ └───┘ └───┘ └───┘ └───┘ with: - CPU: actions happening on the CPU (host-side) - GPU: actions happening on the GPU (device-side) - H2D: host to device transfer - D2H: device to host transfer and: - PR N: preparation of batch N - ->N: host to device transfer of batch N - COMPUTE N: compute step for batch N - <-N: device to host transfer of batch N - UP N: update of batch N You can see that the GPU is almost always busy, except where the █ is. Proper ordering of steps is ensured through the use of CUDA events and streams. r#rDrErFrGrHrIrJr'Nc [RR5(d+[S[RR5<35eSUl[ S5V s/sHn [ UUUUUUUU5PM sn Ul[RRUS9Ul [RRUS9Ul [RRUS9Ul SURSRl SURSRl SURSRl SURSRl URUlgs sn f)Nz>Async batching requires CUDA, but torch.cuda.is_available() = rrorPr )r6rOr  RuntimeError current_pairrUrio_pairsrb h2d_stream d2h_streamrcr r rvrds r,rf#ContinuousBatchingAsyncIOs.__init__sCzz&&((!`EJJD[D[D]Cabc c1X   %     **++6+:**++6+:#jj//v/>26 a  /48 a""126 a  /48 a""1 % 6 6/  s#E7cdURURRR5$r)rrr rr*s r,r1ContinuousBatchingAsyncIOs.get_cumulative_seqlenss&}}T../77NNPPr/rZrrrRrScURURnURRXX4U5 URRR UR 55 gr)rrr rrrinfer_carry_over_ids)r+rZrrrRrSio_pairs r,r0ContinuousBatchingAsyncIOs.prepare_batch_tensorssV-- 1 12-- 2FVa  &&,,T-F-F-HIr/cURURRRnURSUR- RRn[ UR 5Vs/sHnSPM nn[ UR55HupVURU5nUcMXTU'M! [R"U[RS9$s snf)aInfers the ids of the tokens to carry over from batch N to batch N+1. In asynchronous batching mode, we can schedule a request for batch N+1 without knowing the token predicted for that request in batch N. For that reason, we might need to carry over tokens just predicted in batch N before launching the forwar pass of batch N+1. This method computes the ids of the tokens to carry over.r rr) rrr r[rUrvrrgetr6rr|)r+!next_req_id_to_new_token_position!prev_req_id_to_new_token_positionrerrreq_idnew_token_positions r,r&/ContinuousBatchingAsyncIOs.infer_carry_over_idss -1MM$:K:K,L,T,T,q,q),0MM!d>O>O:O,P,X,X,u,u)&+D,A,A&BC&B"&BC ##D#I#I#KLIA!B!F!Fv!N !-5612M||N%++>>Ds5 C"rc0URURnURUR5 URR UR 5 UR RUR 5 URRUS9$)N)r) rrrr  record_eventr rc wait_eventr r)r+rr's r,r+ContinuousBatchingAsyncIOs.get_model_kwargsst-- 1 12##DOO4 $$W%5%56 &&w'7'78  11k1JJr/cURURnURSUR- nURRURRURR4$)aReturns the tensors used inside the generation step that are not inputs to the model forward pass. Those tensors could be retrieved using this object, but it would trigger a recompile if using torch.compile. They are: - output_ids: the output ids of the current batch - prev_output_ids: the output ids of the previous batch, required to carry over outputs tokens of the previous batch to the input tokens of the next batch. - carry_over_ids: a mask representing how to carry over tokens. r )rrr rr)r+r previous_pairs r,r(ContinuousBatchingAsyncIOs.get_cb_kwargssh}}T%6%67  a$*;*;&;<  " " 1 1  # # . .  " " - -  r/rrrcUSU4nUS:gR5nUSURS5nUSURS5nXE-USSU- --US'g)zAs explained in the infer_carry_over_ids method, we might need to carry over tokens just predicted in batch N before launching the forwar pass of batch N+1. This method performs the carry over, and is recorded in CUDA graphs if they are enabled.rrNr )r;ru)r+rrrcarried_over_idscarried_over_masks r,r,ContinuousBatchingAsyncIOs.carry_over_tokenssv+1n+<=+r1668+,?innQ.?@-.A q0AB';ilaRcNc>dd ! r/c\URURRR$r)rrr rr*s r,r%ContinuousBatchingAsyncIOs.output_ids s%}}T../99DDDr/cdURURRR5$r)rrr rr*s r,r$ContinuousBatchingAsyncIOs.get_graphs&}}T../99CCEEr/rchURURRRU5 gr)rrr r)r+rs r,r$ContinuousBatchingAsyncIOs.set_graphs% d''(22< ""7#7#78$$T__5 $$W%5%56 1 11r/cURURnURR5 URR 5$r)rrrrr rrDs r,r/ContinuousBatchingAsyncIOs.prepare_batch_update)s;-- 1 12$$&3355r/cSUlURHnUR5 M URR 5 UR R 5 UR R 5 g)zWReset all state for a new generation session. Used in persistent mode between sessions.rN)rrrr rr!rcrDs r,r ContinuousBatchingAsyncIOs.reset.sS}}G MMO% ##% ##% '')r/)rcrr!r rrvrr)'r1r2r3r4r5r rr6rErlr;r=r rfrr7r9r:rr<rrr&rrrrpropertyrrOrrrrYrrrrr>r0r/r,rrhs*+l','7"'7!'7 '7 [[ '7  '7'7?'7 $'7 '7TQellDellARrTsq#! =*#&G>>jj 6 6  6 rV3V3r3X3XlM*M*r/