pyi (SSKrSSKrSSKrSSKJr SSKJr SSKJr SSK r SSK J s J r SSKJrJrJrJrJrJrJrJr SSKJr SSKJrJr \R8"\5rS rS r S S S S.r!S\S\S4\S4\S4\S44\S4\S\!SS344S.S\S\S44SS.S\S\S44S S.S!.r"Sq#Sq$Sq%Sq&Sq'Sq(Sq)S"S#S$.r*S%S&0r+S[S(\,S-S)\S-S*\-4S+jjr.S,r/S[S(\,S-S)\S-S*\-4S-jjr0S\S(\,S-S*\-4S.jjr1S/r2S]S0jr3S1r4S2\ RjS3\6\ Rj\ Rj\744S4jr8S5\ RjS6\ RjS7\ RjS2\ RjS8\74 S9jr9S:r:S;r;S<r<S]S=\ RjS>\ RjS?\ RjS@\ RzS-4SAjjr>"SBSC\S'SD9r?S^S8\7SE\7SF\-SG\@SH\@S-SI\7S-SJ\-SK\@S-SL\-S-S%\ RjS-SM\7\ R-S-SN\7\ R-S-SO\B\,\-4S-4SPjjrCS_SQ\ RjSR\ RjSS\ RjS2\ RjS-S8\7SF\-SG\@ST\ RjS-SH\@S-SI\7S-SJ\-SK\@S-SL\-S-SU\ RS-SV\ RS-SW\7S-SX\7S-S@\ RzS-SY\,S-4&SZjjrEg)`N)Callable)partial) TypedDict)is_flash_attn_2_availableis_flash_attn_3_availableis_flash_attn_4_availableis_torch_cuda_availableis_torch_mlu_availableis_torch_npu_availableis_torch_xpu_availablelogging)split_attention_implementation)PACKAGE_DISTRIBUTION_MAPPING is_tracingcx[5(d[5(d[5(agSSKJn U"5$)NFr'is_npu_fa2_top_left_aligned_causal_mask)rrr integrations.npu_flash_attentionrrs h/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/modeling_flash_attention_utils.py!flash_attn_supports_top_left_maskr*s, ""&?&A&AE^E`E`Y 2 44c[5=(d= [5=(d, [5=(d [5=(d [ 5$N)r rrr r rris_flash_attn_availabler4s=!# $ $ & $ $ & $ " # $ " # rzkernels-community/flash-attn2z"kernels-community/vllm-flash-attn3zkernels-community/flash-attn4)flash_attention_2flash_attention_3flash_attention_4c[RRS5SL=(a, S[SVs/sHo"R SS5PM sn;$s snf)N flash_attnz flash-attn_- importlibutil find_specrreplaceargskwargspkgs rr.NsR)..:R:RS_:`hl:l;j >Z[g>hi>hs[[c2>hi i;jiAcudamlunpuxpuz+Detect using FlashAttention2 on Ascend NPU.z*Detect using FlashAttention2 (via kernel `rz `) on XPU.)flash_attn_versiongeneral_availability_checkpkg_availability_checksupported_devicescustom_supported_devicesc[RRS5SL=(a, S[SVs/sHo"R SS5PM sn;$s snf)Nflash_attn_interfacez flash-attn-3r#r$r%r*s rr.r.asR)..:R:RSi:jrv:v;v @\]s@tu@t{{34@tu u;vur/)r4r5r6r7cuda_min_major_versionc[RRS5SL=(a, S[SVs/sHo"R SS5PM sn;$s snf)Nr"z flash-attn-4r#r$r%r*s rr.r.isR)..:R:RS_:`hl:l;l @\]i@jk@j{{34@jk k;lkr/ )r r9r> dropout_p window_size)dropoutsliding_windows_auxlearnable_sinkFimplementationattention_wrapperallow_all_kernelsc[5n[5n[5n[[pv[ U5upUS:XaU(dUc)U(a"U(dU(dSSKJn Jn J n SSK J nJ n GO[5(aSSKJn SSKJn SS KJn OUS :XdUcU(aU(d SSKJn Jn J n OUS :Xd UcU(a SS KJn Jn Sn OSS KJn [,R/X5n U(aSU3OU nU "XUS9n[1USS5n [1USS5n [1USS5n U c[3SUS35eU c[4R7SUS35 U c[4R7SUS35 XXU4$)a Lazy loads the respective flash attention implementations. Return: flash_attn_func: The base flash attention function. flash_attn_varlen_func: The flash attention function supporting variable sequence lengths, e.g. for padding-free training. pad_input: The function to pad inputs into one sequence and returning the respective kwargs. unpad_input: The function to unpad outputs based on the kwargs (from pad_input). rNr)flash_attn_funcflash_attn_varlen_funcflash_attn_with_kvcache) pad_input unpad_inputr)npu_flash_attn_func)npu_flash_attn_varlen_func)npu_flash_attn_with_kvcacherr)rKrL)load_and_register_attn_kernelzpaged|rIrKrLrMzJCould not find the currently requested flash attention implementation at `z_`.Make sure that you request a valid kernel from the hub, e.g. `kernels-community/flash-attn2`.z.The loaded flash attention implementation at `z` only supports varlen, i.e. it can only be used with continuous batching and does not support the full functionality for the base transformers generation methods.z` does not support block tables, so the full performances of continuous batching will not be achieved, only the varlen path will be used.)rrr _pad_input _unpad_inputrr"rKrLrMflash_attn.bert_paddingrNrOr rrPrQrRr;flash_attn.cuteintegrations.hub_kernelsrSFLASH_ATTN_KERNEL_FALLBACKgetgetattr ValueErrorloggerwarning)rGrHrIis_fa2is_fa3is_fa4rNrOis_pagedrKrLrMrS kernel_repokernel_implementationkernels r _lazy_importsrgs' (F & (F & (F'{=nMH--&6&__BB  ! ! ]jl 0 0^5KPV_e m m 2 2~7MRX O&* # P588XKAIf^,<$={ !2%L]F&f.?FO%,V5Mt%T "&-f6OQU&V #%- `ao`pqtt&D^DTU@@ '.D^DTU 4KXc ccrc6[R"U5Rn[R"[5Rn0nUH@n[R XD5nXQ;X5'[ R XD5=oe:wdM:Xa;X6'MB [[US9$)a Depending on the version and kernel some features are not supported. Due to limitations in `torch.compile`, we opt to statically type which (optional) kwarg parameters are supported within `_process_flash_attention_kwargs`. NOTE: While all supported kwargs are marked as `True`, everything else is marked as `False`. This might be confusing for kwargs that we use in any case, e.g. `is_causal`. )supports_mapping)inspect signature parameters_process_flash_attention_kwargs_hf_api_to_flash_mappingr[_flash_api_alternative_namesr)flash_functionflash_parametersprocess_parametersriparamfa_paramfa_alternative_names r_lazy_define_process_functionrvs((8CC **+JKVV#+//=%-%A"#?#C#CE#Q Q  ^4G4[  1 $ 2EU VVrcUc[c [S5eUb+[U:wa!Uq[XUS9uqqqqq[[5q [[[ [ [4[4$)z Lazily import flash attention and return the respective functions + flags. NOTE: For fullgraph, this needs to be called before compile, while no fullgraph can work without preloading. See `load_and_register_attn_kernel` in `integrations.hub_kernels`. zGCould not find any flash attn implementation based on your environment.rT) _loaded_implementationr]rg _flash_fn_flash_varlen_fn_flash_with_kvcache_fn_pad_fn _unpad_fnrv_process_flash_kwargs_fn)rGrHrIs rlazy_import_flash_attentionrsx"8"@bcc!&<&N!/R_ ARS O #%;Wi$AAQ#R ')?) TVn nnrc8SSKJn [XUS9uup4n nXE4$)za Same as `lazy_import_flash_attention` but explicitly wrapping it with the paged implementation. r)paged_attention_forward)rHrI)integrations.flash_pagedrr)rGrIrr#rLflash_attn_with_kvcache_fns r!lazy_import_paged_flash_attentionrs3BGbUfHDAQ :Aq1 " ==rcJUR"S/URSSQ76nX!$)z A local implementation of the PyTorch indexing operation `tensor[indices]` on the first axis, after flattening the first two dimensions of the tensor. This is functionally equivalent to FA2's `index_first_axis` and replaces the need to import it. r N)reshapeshape)tensorindicesreshaped_tensors r_index_first_axisr s+nnR;&,,qr*:;O  ##rcUbX-OUnURS[RS9nURS[RS9n[R"UR 5SS9R 5nUR 5n[ R"[R"US[RS9S5n[X5UUUU4$)a unpad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. rdimdtypeFas_tuplerrr) sumtorchint32nonzeroflattenmaxFpadcumsumr) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchrmax_seqlen_in_batch cu_seqlenss rrVrVs 3>2I-~I }}5;;}?*..2U[[.ImmI--/%@HHJG*..0u||$4!5;;OQWXJ -1  rcURSSn[R"X#-/UQ7URURS.6nXU'UR "X#/UQ76$)a pad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) rN)devicer)rrzerosrrview)rrbatchseqlenroutputs rrUrU8sZ   ab !C [[%. hC h 8L8LTaTgTg hF#7O ;;u +s ++rrreturnc<URS[RS9n[R"UR 5SS9R 5nUR 5n[ R"[R"US[RS9S5nUUU4$)aA Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. rrFrrr) rrrrrrrrr)rrrrrs r_get_unpad_datarKs &))b )DmmN224uEMMOG*..0u||$4!5;;OQWXJ r query_layer key_layer value_layer query_lengthc[U5upgnURSURS=n :a!USS2SU 2SS2SS24USS2SU 2SS2SS24p!URupp[X5n[X&5nXK:Xa[X5nUnUnUnOhUS:XaJSn[R"U S-[R UR S9nUSSnURS5nOUSS2U*S24nU"X5tnnpnUUUUX4X44$)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. unpad_input_func: The function to use for unpadding the input tensors. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rrNrr)rrrrarangerrsqueeze)rrrrrunpad_input_func indices_k cu_seqlens_kmax_seqlen_in_batch_kseq_len batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_qr#s r _upad_inputrfsER6E^5T2I2q(<((9:JZ[fJwG Y a  $ 6  rc[RURS.nURS5nUS:HR 5R S5n[R "UR"S0UD6[R"UR540UD645nUnUR5R5nUnX44XV44$)a This function returns all the necessary kwargs to call `flash_attn_varlen_func` extracted from position_ids. Arguments: position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rrrr) rrrrrrcattorsizediffr) position_ids tensor_kwargsr cu_seq_lens_q cu_seq_lens_k max_length_q max_length_ks r#prepare_fa_kwargs_from_position_idsrs $kk\5H5HIM''+L"++-2226III LL )= ) LL**, > > M "M !%%'++-LL  )L+G GGrcUR5RSURS5URS55nUR5RSURS5URS55nUR5RSURS5URS55n[U5uupEupgXX$U4Xg44$)ai This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cumulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cumulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). r) contiguousrrr)querykeyvaluerrrrrs r_prepare_from_posidsrs:     # #B 2 2 GE ..   CHHRL#((2, ?C     # #B 2 2 GECfgsCt@"]$@\ }= ?[ \\rcUcg[R"URSURS9UR 5-nUS:H=(a. X - R 5R 5R5$)a Check the position ids whether packed sequences are indicated or not 1. Position ids exist 2. Flattened sequences only are supported 3. Compile-friendly `not (torch.diff(position_ids, dim=-1) >= 0).all()`, i.e. we have multiple increasing sequences Fr)r)rrrrminabsrbool)rrincreasing_position_sequencess r_is_packed_sequencersq  \''*<3F3FG,JZJZJ\\" ? ` = LQQSWWY^^``rqkv target_dtypecU(aiUR[R:XaK[R SUS35 UR U5UR U5UR U5p!nXU4$)aM PEFT usually casts the layer norms in float32 for training stability reasons therefore the input hidden states gets silently casted in float32. Hence, we need cast them back in float16 / bfloat16 just to be sure everything works as expected. This might slowdown training & inference so it is recommended to not cast the LayerNorms! zCasting fp32 inputs back to z for flash-attn compatibility.)rrfloat32r^ warning_oncer)rrrrs rfa_peft_integration_checkrs^5==0:<.Hfgh$$|$add<&8!$$|:La 7Nrc\rSrSr%Sr\R S-\S'\R S-\S'\S-\S'\S-\S'Sr g) FlashAttentionKwargsi#a Keyword arguments for Flash Attention with Compile. Attributes: cu_seq_lens_q (`torch.LongTensor`, *optional*) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`, *optional*) Gets cumulative sequence length for key state. max_length_q (`int`, *optional*): Maximum sequence length for query state. max_length_k (`int`, *optional*): Maximum sequence length for key state. Nrrrrr) __name__ __module__ __qualname____firstlineno____doc__r LongTensor__annotations__int__static_attributes__rrrrr#s? ##d**##d****rr)total key_length is_causalrC softmax_scalerDuse_top_left_masksoftcap deterministic max_seqlen_q max_seqlen_kric  U=(a U=(a US:H(+US.nU S(aX>S'U S(aUbX:a US- US- 4US'U S(a"UbUO[R"SS5S:HUS'U S (aUbX~S 'U S =n(d U S (aU bU(aXS 'OXS 'XLnU S (a<U b9[U [5(d [ U 5(aU R 5n XS 'U S (aOU bLU(a US bUS n O5[U [5(d [ U 5(aU R 5n XS 'U$)a Returns a set of kwargs that are passed down to the according flash attention function based on requested features and whether it is supported - depends on the version and kernel implementation which is dynamically configured at `lazy_import_flash_attention`. The (un)supported features can be inspected in `supports_mapping`, see `_lazy_define_process_function` for more details. Args: query_length (`int`): Length of the query states key_length (`int`): Length of the key states is_causal (`bool`): Whether we perform causal (decoder) attention or full attention. dropout (`float`): Attention dropout. softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to `1 / sqrt(head_dim)`. sliding_window (`int`, *optional*): The size of the sliding window, i.e. we look at a max of `sliding_window` tokens back. use_top_left_mask (`bool`): Deprecated behavior of older versions of flash attention requiring different masking. softcap (`float`, *optional*): Softcap for the attention logits, used e.g. in gemma2. deterministic (`bool`, *optional*): Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled. s_aux (`torch.Tensor`, *optional*): Attention sink auxiliary that adds a `bias` to the attention calculation via an additional head. max_seqlen_q (`Union[int, torch.IntTensor]`, *optional*): The maximum sequence length in the query tensor during a varlen forward. max_seqlen_k (`Union[int, torch.IntTensor]`, *optional*): The maximum sequence length in the key/value tensor during a varlen forward. Return: flash_kwargs (`dict`): A dict of kwargs that are requested and supported. r)causalrrArBrFLASH_ATTENTION_DETERMINISTIC01rrErFrr)osgetenv isinstancerritem)rrrrCrrDrrrrErrrir, flash_kwargslegacy_sink_paramsame_max_seqlens rrmrm8syhM%6%L<1;L M&L  $$+[! &>+E*Je (6'9>A;M&N ]#(*6MBIIFegj