!Ei( 0%SrSSKrSSKr/r\\\S'S\RS\4Sjr S\RS\ S\R4S jr S \ S \ S-S\ 4S jr SS \RS\RS\RS\RS-SS4 Sjjrg)zCDefines utilities for interacting with scaled_dot_product_attentionN__all__tensorsreturnc&[SU55$)z0Returns True if any of the tensors requires gradc38# UHoRv M g7f)N) requires_grad).0ts V/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/nn/attention/_utils.py '_input_requires_grad..s01s)any)rs r _input_requires_gradr s 00 00 inpt_tensorog_sizecBURS5U:wa USSU24$U$)z'Handles the unpad of the last dimension.N)size)rrs r _postprocess_flash_outputrs.w&3=)) r head_dim_sizescalec>UbU$S[R"U5- $)z For FlashAttention we pad the head dimension to be a multiple of 8 so we need to scale the output by the original head size and not the padded. g?)mathsqrt)rrs r _calculate_scalers#   =) ))rquerykeyvalue attn_maskc U(dgURUR:wdURUR:wa3[SURSURSURS35eURUR:wdURUR:wa3[SURSURSURS35eUR5S:d(UR5S:dUR5S:a?[S UR5S UR5S UR5S35eg) NzLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: z instead.zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: zUExpected query, key, and value to all be at least 2 dimensional, but got query.dim: z , key.dim: z and value.dim: )dtype ValueErrordevicedim)rrrr dropout_p is_causalr allow_lowp_kvs r _validate_sdpa_inputr*"s%  ;;#)) #u{{ekk'A((- }M#))M$$)KK= ;   ||szz!U\\U\\%A%%*\\N. M!!&i 9   yy{Q#'')a-599;?cyy{m;swwyk1A%))+i Y  ,;r)NgFNF)__doc__rtorchrliststr__annotations__Tensorboolrintrfloatrr*rr r5sI c15<<1D1 5<<#%,,*C* **&*  <<   << ||d"    r