>:j :SSKrSSKJr SSKrSSKJr S\R S\R 4SjrSrS\RS\R 4Sjr SS\RS \\ \RS \\R S\R 4S jjr S \S\4S jrg)N)OptionalxreturncUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)a Rotate half the hidden dims of the input. This function was duplicated verbatim from: https://github.com/huggingface/transformers/blob/1de8ce9ee1191ba761a593ac15d9ccbf5851bfc5/src/transformers/models/llama/modeling_llama.py#L126 This was done to eliminate the Llama transformers implementation as a dependency of this file. Note that some other functions were also adapted from the transformers implementation but were modified. .Ndim)shapetorchcat)rx1x2s ^/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/adaption_prompt/utils.pyllama_rotate_halfrs\ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''c[UR5S:XaUSS2SSS2S4nURSURSSURS5n[R"URURSSSS5SU5n[R"URURSSSS5SU5nO&XR S5nX#R S5nX-[ U5U--nU$)a Apply rotary position embedding to query states in the Llama model. This function was adapted from: https://github.com/huggingface/transformers/blob/1de8ce9ee1191ba761a593ac15d9ccbf5851bfc5/src/transformers/models/llama/modeling_llama.py#L133 It was modified to remove unnecessary processing of key states. The method is compatible with transformers <= 4.34.2 and also with the latest version (>=4.35). Nrr)lenr repeatr gather unsqueezer)qcossin position_idsgather_indicesq_embeds rllama_apply_rotary_pos_embr!$s 399~%aq$&67'..q#))A,399Q<Pll3::n&:&:1&=q!QGN[ll3::n&:&:1&=q!QGN[))!,))!,w,Q/#56G Nrmodelc URS5nURS5nURS5nUR5upVnURRnUR U5R XVXR 5RSS5n URRURR-n URU5R XVX-UR 5RSS5n Un UbG[U[5(aXSRS- n OXRUR 5- n SU;a<USupU R#S5n UR#S5nX-[%U 5U--$S[&R("UR*R,5R.;aUR+XS 9up[1XX5$SnUctUc#[2R4"XfU-U R6S 9nO=UR9X`R 5n[2R4"XU-U R6S 9nUR#S5nSU0nS [&R("UR*R,5R.;aXo-US 'UR*"U 40UD6up[;U R5S :Xa"U R#S5n UR#S5nX-[%U 5U--$) a Compute query states for Llama models specifically. They need to be recomputed as the forward() method of the original LlamaModel in the transformers library does not return them. See the related discussion in the PR: https://github.com/huggingface/peft/pull/268 hidden_statesrpast_key_valuerrrposition_embeddings)seq_len)devicer(r)getsizeconfignum_attention_headsq_projviewhead_dim transposek_proj in_features out_featuresv_proj isinstancetupler get_seq_length layer_idxrrinspect signature rotary_embforward parametersr!r aranger)get_usable_lengthr)r"kwargsr$rr%bszq_len_ num_heads query_statesfactor value_statesr(rrpast_seen_tokensnew_cache_positionsrotary_emb_kwargss rllama_compute_query_statesrL=s JJ/M::n-LZZ 01N!&&(MC 00I<< .33C >>ZddefhijL \\ % %)B)B BF<< .33CATW\WeWefppqrtuvLG! ne , , a(..r2 2G 44U__E EG&/0mmAmmA"'8'F'LMMW..u/?/?/G/GHSSS##L#B),SOO  !"',,uemLL_L_"` -??W "',,/?TYAYbnbubu"v *44Q7 '6G%%e&6&6&>&>?JJJ','?)$ B0ABHC 399~mmAmmA  #4\#BS#H IIrr$encoder_hidden_statesctUbF[US5(d#[SURRS35eUR U5nO-UR U5R URSS9un n/URSSQSPURP7nURU5RSS5nU$) z Compute query states for GPT2 models. They need to be recomputed as the forward() method of the GPT@ in the transformers library does not return them. See the related discussion in the PR: Nq_attnzIf `z` is used as cross attention, the weights `q_attn` must be defined. Please make sure to instantiate it with `GPT2Attention(..., is_cross_attention=True)`.rr rr) hasattr ValueError __class____name__rOc_attnsplit split_sizer r0r/r1)r"r$rMrFrDshape_qs rgpt2_compute_query_statesrXs(uh''u//01ij ||M2 "\\-8>>u?O?OUV>W a< ""3B'<rhs  ( (%,, (2?Jbii?Jell?JJ59 99E%"3"345$ELL1 \\ 2999r