>:jJDSSKrSSKJrJr SSKJrJrJr SSKJ r SSK J r J r SSK J r SSKJr SSKJrJr SSKrSSKJr SS KJr SS KJr SS KJrJr SS KJr SS KJ r SSK!J"r"J#r# SSK$J%r% SSK&J'r'J(r(J)r)J*r* \'\)\*4r+"SS5r,"SS\,5r-"SS\,5r.S\/S\/4Sjr0S\RbRdS\Rf4Sjr4S\\5\Rf44Sjr6S\%4S jr7S\Rp4S!jr9S"r:S\RbRdS#\S\\%S$\S%\\S&\\S'\\\/\5\4S4S(\;S)\;S\/4S*jr!((*I'00#i.1Y;CTCTUVUWCX1YZL3>/;,,Q/ 0HMZGXYGX!u// =GXOY OOO-D-D-F GBE_A`aA`vet}A`Oa99_!4 4Zbs  FF )r&r"r'r )NT)__name__ __module__ __qualname____firstlineno____doc__strr rboolr* staticmethodr5r6r%no_gradrCr___static_attributes__r,r)rr)sx7;*.    "*(!3  $(   RWR^R^   ]]_WWr,rc^\rSrSrSrS\S\\\R44U4Sjjr \R"5S5r Sr U=r$)SVDHookja A forward hook for calculating incremental SVD on layer inputs. The hook is designed to be registered to a PyTorch module using the `register_forward_hook` method. This hook performs a step of incremental Singular Value Decomposition (SVD) on the inputs of a specified layer during the forward pass of a neural network. The hook also tracks convergence of the computed components using cosine similarity between the current and previous components. Args: name (str): Name of the layer to which this hook is attached. n_components (int): Number of principal components to compute. sim_thresh (Union[float, torch.Tensor]): Similarity threshold for convergence. prepare_layer_inputs_fn (Optional[Callable]): Function to prepare layer inputs for SVD. n_components sim_threshc >[TU]"S0UD6 XlX l[ U[ R 5(ay[UR5S:a`URS5U:H=(d URS5S:Hn[UR5S:HnU(aU(d [S5e[USSSS9Ul SUl [ R"U4[ RS9Ulg) Nrrz`if sim_thresh is a tensor with more than 0 dimensions it must have shape (n_components,) or (1,)T*)ror lowrank lowrank_seed)rGrk)superr*rorpr4r5r6lenrMr=r9rsvdr'zerosrg converged)r(rorpbase_class_kwargscheck1check2 __class__s r)r*SVDHook.__init__zs -,-($ j%,, / /C 8H8H4IA4M__Q'<7R:??1;MQR;RF))*a/Fv v"%    l_EJJGr,cSn[URS5(a2URRR5R 5nUR U5nUR U5nURS5UR:a'[SURSURS35 gURRUR[R55 UcgURRn[UR 5S:Xa$UR#SS5nUR#SS5n[R$R&R)Xd5nXpR*:Ulg)N components_rzskipping SVD for z because there are less than z examplesrr3)hasattrrwrclonedetachrCr_r=roprintr partial_fittor5float32rvrMreshapenn functionalcosine_similarityrpry)r(modelinputoutputprevious_componentsstates componentssims r)__call__SVDHook.__call__s0" 488] + +"&(("6"6"<"<">"E"E"G **51))&1 ;;q>D-- - %dii[0MdN_N_M``ij k  VYYu}}56  & XX)) z A %#++Ar2J"5"="=a"D hh!!33JT/r,)ryr'rorprw)rarbrcrdreintrfloatr5r6r*rirrj __classcell__r}s@r)rmrmjsJ HH%-.H2 ]]_00r,rmcj^\rSrSrSrU4Sjr\S5r\R"5S5r Sr U=r $)HashHooka A forward hook for hashing layer inputs. The hook is designed to be registered to a PyTorch module using the `register_forward_hook` method. This hook hashes the inputs of a specified layer during the forward pass of a neural network and stores the hash values for later analysis or comparison. Args: name (str): Name of the layer to which this hook is attached. hashed_inputs (list): List of hashed inputs. prepare_layer_inputs_fn (Optional[Callable]): Function to prepare layer inputs for hashing. c 4>[TU]"S0UD6 /UlgNrk)rur* hashed_inputs)r(rzr}s r)r*HashHook.__init__s -,-r,cd[[URS5R555$)Nr3)hashr7r<rP)rLs r)hash_fnHashHook.hash_fns#E&++b/002344r,cURU5nURU5nURRUR UR 555 gr$)rCr_rappendrcpu)r(rrrxs r)rHashHook.__call__sE  % %e ,  $ $Q ' !!$,,quuw"78r,)r) rarbrcrdrer*rhrr5rirrjrrs@r)rrs;  55 ]]_99r,r dictionaryr-c[[5nUR5Hup#XRU5 M UR5VVs0sHup#[ U5S:dMX#_M snn$s snnf)a& Find keys in a dictionary that have the same value. This function takes a dictionary and returns a new dictionary containing keys that have the same value. The keys in the output dictionary are the values from the input dictionary, and the values are lists of keys that share the same value. r)rr8itemsrrv)r value_dictkvs r)find_equal_valuesrsfT"J  " Q#'--/ >/TQ3q6A:DAD/ >> >s A2(A2rc[UR5Vs1sH*oRRS:wdMURiM, sn5n[ U5S:a[ R "SU35 gUS$s snf)zY Get the device of the model's parameters. Useful if some parameters are on meta device. metarz8Could not determine device, model has multiple devices: Nr)r8 parametersrFr:rvwarningswarn)rpdevicess r)get_device_with_meta_paramsrspe&6&6&8T&8HHMMVA>rFc ^[US5(aURT5$[U[5(a?[ U5"UR 5VVs0sHup#U[ UT5_M snn5$[U[[45(a[ U5"U4SjU55$[R"S[ U5S35 U$s snnf)zC Move the inputs to the specified device. Adapted from hf.Trainer. rc3<># UHn[UT5v M g7fr$)move_inputs_to_device).0rrFs r) (move_inputs_to_device..sMf1!V<r,c[U[R5(aO<[U[[45(aUSnO[ S[ U5SUS35eXRR5$)a if not all items in the input should be used for SVD, this function can be used to get the indices of the items that should be used. Attributes: layer_input (torch.Tensor): The layer inputs. model_input (torch.Tensor): The model inputs or if `prepare_model_inputs_fn` is not None the output of this function. layer_name (str): The name of the layer. Returns: torch.Tensor: The input to the SVD. rr/r0r1) r4r5r6r7r8r9r:Tunbindr>s r))prepare_layer_inputs_fn_language_modelingrsx+u||,, K% / /!!n %d;&7%88^_i^jk> >  }}++- ..r,cU"S0UD6$rrk)rrs r)forward_fn_dictrs ?6?r, dataloadertarget_module_check_fn forward_fnprepare_model_inputs_fnr!r"show_progress_barc ^0Sn [U5S:Xa [S5e[R"5(aU(a[R "S5 Sn UR Rn X:a9[SUR555n XR-n [X5n URn[U5nUR5 [[!U55nUb [#UU5nUb U"UU5nO [%U5n0n0m0SnUR'5HunnU"UU5(dM[)U[*5(aUR-US5nOUn[/UUUS9nUUlUR3U5nUU4UU'UR4R7[9UR4R;5U5UR5n[=UU -5T0U'UU- nM [)U[*5(a+[U5S:a[SUR;535eU"UU5 UR?5VVs0sHunnUUSR@S_M nnn[EU5RG55nUVVs0sHnUS SH nUUS_M M n nnUH&n![U04S jU!55n"U!Hn#U"T0U#'M M( [CUR;55HnUR-U5unnURI5 UU ;aM/[KT0UUR RLUURNUS 9nURQU5nUR3U5nUU4UU'M 0[S[UUR;5UR;555EU En$U(aS[R"5(a[RV"5S:Xa [Y[![[U55SS S 9n%Sn&O[![[U55n%S n&UR;5Vs0sHnUS _M n'nT0R]5n(U%GHnUb [#UU5nUb U"UU5nO [%U5n[CUR;55HnUUunn[^R`"URbSU(U5n)U'U(d'U)(a U(aURI5 SnSU'U'MbU'U(a.U)(d'URQU5nUR3U5nS U'U'UUlUU4UU'M U&(ag[CU'RG55U RG5Vs/sHnU'UPM sn-n*U%Re[gU*5S[U*5S35 [aU'RG55(a OBU"UU5 [aSURG555(dGMU "UU$U UT05n(GM UR'5V#V+Vs1sHAun#n+U+RhRG5Hn[)U[j5(dMU#iM MC n,n+n#n[U,5S:a[SU,S35e0n-U(R?5Hunn.UU$USn[^R`"URbSU.5(d[SUSU.S35eURlRnSU.n/UR Rp(a:U/URlRrSU.Ru5RwSS 5-n/U/U-U'M URyU5 Ub5U-R?5VVs0sHunnUUR{U5_M n-nnU-$s snnfs snnfs snfs snfs snn+n#fs snnf)Nc xUR5VVs0sH#upVXVSRRSXE_M% nnn[UR5VVV s/sHupXXxHoU 4PM M sn nn6up[R "U 5R SS9n [U SUV s/sHoU PM sn 5nUR5Vs0sHoUURUS5_M nnUR5Hup_XXnnUU:aMUUsX'X'M U$s snnfs sn nnfs sn fs snf)NrT) descending) rrwexplained_variance_ratio_rWr5stackargsortrkeysr)hookslayer_hook_mapequal_inputs_map rank_budgetmax_componentsrhexp_varsr crvaluesidxicountsk_hookrank rank_hooks r)_get_rank_distribution3_get_eva_state_dict.._get_rank_distribution.s8[`[f[f[hi[hSWSTAtxx99:MN3G3G3Ib3IS[SaaVSaV3Ibc kk&!))T):3| +<=+/=/B/B/DE/D!VZZ1%%/DE)//1IA$i)DD (,i %FNFI 2  jb=Es*D%D+ ,D2D7rzdataloader is emptyztorch.distributed is initialized and `gather_distributed_inputs` is True, therefore EVA initialization will gather tensors from all ranks. Ensure the model does not receive the same inputs on different ranks.ic3L# UHn[UR5v M g7fr$)rRrM)rrs r)r&_get_eva_state_dict..Ls?,>qc!''ll,>s"$)r r!r"zaprepare_layer_inputs_fn is a mapping but the following module names were not found in the model: rc3.># UH nTUv M g7fr$rk)rnrs r)rrs95aq)5s)rorpr r!r"F)positionleaveT/z layers have convergedc3T# UHn[USRS5v M g7f)rrN)rrw)rrs r)rrs"L^71Q488]33^s&(z?Found active hooks added by EVA that weren't properly removed: zH. Please report this issue at https://github.com/huggingface/peft/issueszLayer z) has not converged but was assigned rank r3)>rvr9rIrJrrrrhorRrrmintrainingrevalnextiterrr named_modulesr4rpoprr'register_forward_hook rank_patternrrrroundrrr8rrremovermtaur& get_submodulerrWget_rankrr r r5allryset_descriptionsum_forward_hooksrrwrwhitensingular_values_sqrtrtrainr)1rrrrrrr!r"rr rho_thresholdrmax_dimrho_ceilrrFrmodel_inputs_for_hooksrrr modulefnhookhandle layer_rankrr hash_dict equal_inputsrvvrnames max_valuerrpbaruse_tqdmconvergence_dict rank_distrylayer_convergedmremaining_hookseva_state_dictrurs1 @r)_get_eva_state_dictr(!s}  :!.// !:  T M  $ $C ?E,<,<,>??mm+# ~~H ( /F JJL$z" #F &vv6*!8!M!)&!1 ENK++- f%dF33  -w 7 7(,,T48B(BT2Yrs1--d3Vnd  --11 K4499;T BKMM  %Z#%56tz! . )733L*6G,Q122AaD,G9599 A )N1  UZZ\"yy f  # # '-"--11$($A$A&?  $$T*--d3Vnd #SSuzz|<=RAQRN$"5"5"7"74==?a;ODz*+auEE*%&*/**,7,Q5,7##%I  *66:F " .%_ 1:_/ __r& adapter_namec[URU5n/n/n/n0n0nUR5GH,upU RSS5n [ U [ 5(dUR U 5 M@URR[URR5U 5UR5n URR[URR5U 5UR5n X;GaURU 5nURS5nUS:Xa-[!X 5unnn[#UUU R%55 GM+X:wa"UR&R((aXU - -n X:wd1U R*UR,R.R0S:XaSUlU R5X/XS9 U R*UR,R7U5 UR U 5 O*SUlU R5X,XS9 UR U 5 U nXR:waXU 'XR:wdGM(XU 'GM/ XT-n[9U5[::a [=XV5nXPRUlXpRUlXRUl U(a![@RB"SUS [D35 gg) Nzbase_model.model.rreva)r)r lora_alphaconfigTzuthe following layers were initialized with init_lora_weights=True because they were not found in the eva state_dict: z@ currently the following lora modules are not supported by EVA: )#r rrreplacer4rrrrrrr alpha_patternr-rr=rsetattrget_base_layerradjust_scaling_factorslora_AweightrFr:init_lora_weights update_layercopy_rvrrtarget_modulesrrUNSUPPORTED_LORA_MODULES)rr&r)rmissing_eva_initsnew_target_modulesother_module_namesrr0r rname_in_base_modelralphawnew_rankparentr] target_names r)_load_eva_state_dictrDs u((67KLM++- !\\*=rB&),,  % %&8 9   $ $ ( (9Q9Q9V9V9XZ^)_alanan o))-- K55::.target_module_check_fn_peft_modelasv|,aZ5a1aar,cSnURb [X 5n[U[RR [ 45=(a U$)z9check if a module is an adapter module via target_modulesT)r9rr4r5rLinearr)r rris_target_modules r)target_module_check_fn_default:get_eva_state_dict..target_module_check_fn_defaultes?  % % 19+L &588??F";<QAQQr,rNz3peft_config is required if model is not a PeftModel)rI)r) rrrrrrr!r"r)rrr9disable_adapterr r:rr()rrrrrr!r)r"rrJrO is_peft_modelctxrr&s r)get_eva_state_dictrT,sjbRE=1M,'' 5  NOO##%!( -H`" m!()G!a ,!##9!$;$;&?/      s 6B B!c .[US5(d [S5e[UR5S:a [S5eURUR S:wa [S5eUcUc [S5e[ UUUUUUUUS 9n[XU5 g) a Initialize the weights of the LoRA layers using the EVA method. This function initializes the weights of the LoRA layers using the EVA method. It computes the SVD for each adapter layer and updates the weights accordingly. Args: model (PeftModel): The peft model to compute the SVD for. dataloader (Optional[Iterable]): The dataloader to use for the forward pass. If None, eva_state_dict needs to be provided. eva_state_dict (Optional[dict]): The state_dict to load into the model. If None, a dataloader needs to be provided and the state_dict will be computed using `get_eva_state_dict`. forward_fn (Callable): The forward function to use for the forward pass. Takes two arguments: `model` and `inputs`. Default behavior is `return model(**inputs)` prepare_model_inputs_fn (Optional[Callable]): This function receives the model inputs and the peft_config and passes the output to `prepare_layer_inputs_fn`. Can be used to modify the input to the SVD computation based on the original model inputs. For example for language modeling the attention mask is used to determine which indices are padding tokens and should not be used for SVD. Any function defined here expects two arguments: `model_input` and `peft_config`. `peft.tuners.lora.eva.prepare_model_inputs_fn_language_modeling` is used by default. prepare_layer_inputs_fn (Union[Callable, Dict[str, Callable], None]): This function receives the layer inputs, the model inputs (potentially modified by `prepare_model_inputs_fn`) and the name of the layer and returns the inputs that should be used for SVD for that particular layer. Any custom function defined here expects three arguments: `layer_input`, `model_input`, and `layer_name` and should return a 2d tensor. The default logic can be found in peft.tuners.lora.eva.prepare_layer_inputs_fn_language_modeling and works for language modeling. In this case model_inputs is the mask used to determine which indices should be used for SVD (created by `prepare_model_inputs_fn_language_modeling`). adapter_name (str): The name of the adapter to initialize the weights for. gather_distributed_inputs (bool): Whether to gather the layer inputs from all ranks. Default is True meaning in a distributed setting the layer inputs will be gathered from all ranks for the SVD computation. For non-distributed settings this argument is ignored. Set to False if you are using a non-distributed dataloader in a distributed setting. show_progress_bar (bool): Whether to show a progress bar. Default is True. Returns: model (torch.nn.Module): The model with the initialized LoRA weights. rzmodel must be a PeftModelrzO`initialize_lora_eva_weights` currently only works with a single active adapterr,zM`initialize_lora_eva_weights` can only be used with `init_lora_weights='eva'`Nz8dataloader is required if eva_state_dict is not provided)rrrrr!r)r"r)rr9rvactive_adaptersrr6rTrD) rrr&rrr!r)r"rs r)initialize_lora_eva_weightsrWsj 5- ( (455 5 !A%jkk &88EAhii  WX X+!!$;$;%&?/   =r,)Ar collectionsrrcollections.abcrrr contextlibrr r functoolsr itertoolsr typingr rr5torch.distributed distributedrIrtransformers.pytorch_utilsrpeft.tuners.tuners_utilsrrpeft.utils.constantsrpeft.utils.incremental_pcarpeft.utils.otherrrr.rlayerrrrrr:rrmrrrrModulerFrrfrrr6rrrgr(rDrirTrWrkr,r)rgs?,77"" -]D5=DD&'97C>>B>0e>0F9u98 ?$ ?4 ?uxx5<< %U\\0A*B    /W\WcWc/8E 88??EE*%E% E " E &h/ E#8T#x--@$#FGE $EE EP@ 88??@ @ @ F)-%42[Js!&*"^ 88??^^*%^" ^ &h/ ^ #8T#x--@$#FG ^^ $^^ ^^B&*%)%42[Js!&*"O> 88??O>"O>TNO>" O> &h/ O> #8T#x--@$#FG O>O> $O>O>O>r,