>:je>SSKrSSKrSSKJr SSKrSSKJr SSKJr SSK J r SSK J r "SS\ 5r "SS \R\ 5rg) N)Optional)Conv1D)BaseTunerLayer) transposec^\rSrSrSrSrS\R4SjrS S\ S\ S\ 4S jjr S r g ) GraloraLayer) gralora_A gralora_Bgralora_A_generalgralora_B_general)rhybrid_ralphascalinggralora_dropout base_layerc Xl0Ul0Ul0Ul0Ul0Ul[ R"05Ul[ R"05Ul [ R"05Ul [ R"05Ul [ R"05Ul SUl/UlUR!5n[#U[ R$5(aUR&UR(pCOv[#U[*5(aJ[-UR.S5(aUR.R0OUR.R2up4O[5S[7U535eX0lX@lX lg)NFds_shapezUnsupported layer type )rrr gralora_krrnn ModuleDictr ParameterDictr r r r _disable_adaptersmerged_adaptersget_base_layer isinstanceLinear in_features out_featuresrhasattrweightrshapeNotImplementedErrortypekwargs)selfrr&rr s V/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/gralora/layer.py__init__GraloraLayer.__init__ s7$   !}}R0))"-))"-!#r!2!#r!2"'!((* j")) , ,(2(>(> @W@W  F + +.5j6G6G.T.T !!**ZdZkZkZqZq &K&(?Z@P?Q&RS S&( rr init_weightsc  US::a[SU35eUS:a[SU35eX0RU'X@RU'X`RU'XpRU'US:a[ R "US9n O[ R"5n URR[ R"X055 URU-S:wa[SURSU35eURU-S:wa[SURSU35eURU-n URU-n Un /n /n[U5GH7n[ R"[R "X55n[ R"[R "X55nU(aR[ R"R%U[&R("S 5S 9 [ R"R+U5 Od[ R"R%U[&R("S 5S 9 [ R"R%U[&R("S 5S 9 U R-U5 UR-U5 GM: [R."[1U 5SS 9n U R3S S 5R55n [R."[1U5SS 9nUR3S S 5R55nUS:Ga%[ R6"URUSS9n[ R6"XpRSS9nU(af[ R"R%UR8[&R("S 5S 9 [ R"R+UR85 O[ R"R%UR8[&R("S 5S 9 [ R"R%UR8[&R("S 5S 9 O*[ R"5n[ R"5nXR:U'XR<U'UUR>U'UUR@U'X l!XLU-- URDU'URGU5 URIURJ5 g)Nrz?`r` should be a positive integer value but the value passed is zJ`hybrid_r` should be a non-negative integer value but the value passed is )pz6in_features should be divisible by gralora_k, but got z and z7out_features should be divisible by gralora_k, but got )a)dimF)bias)& ValueErrorrrrrrDropoutIdentityrupdaterrr range Parametertorchemptyinitkaiming_uniform_mathsqrtzeros_appendstacktupler contiguousrr"r r r r module_namer%_move_adapter_to_device_of_base_layer set_adapteractive_adapters)r' adapter_namerGrrrrrr,gralora_dropout_layersubblock_in_featuressubblock_out_features gralora_rr r _new_Anew_Bgeneral_gralora_Ageneral_gralora_Bs r( update_layerGraloraLayer.update_layer@s 6^_`^abc c \ijristu u |#( < '0|$&. l# S $&JJ$A !$&KKM ! ##BMM<2W$XY   i '1 ,HIYIYHZZ_`i_jk    y (A -I$J[J[I\\abkalm  $//9< $ 1 1Y >   y!ALLY!MNELL-B!NOE(($))A,(?u%(($))A,(?(($))A,(?   U #   U #"KKi 0a8 ''1-88: KKi 0a8 ''1-88: a< " $*:*:H5 Q  " (4E4EE R (():)A)ATYYq\(R0778(():)A)ATYYq\(R(():)A)ATYYq\(R "  " '0|$'0|$/@|,/@|,&%*(.B%C \" 22<@ --.r+)rrrr r r r rrrrr&rrGr rrN)r4rT) __name__ __module__ __qualname____firstlineno__adapter_layer_namesother_param_namesrModuler)intboolrU__static_attributes__r+r(rrsW^P299N!Y/Y/Y/Y/Y/r+rc^\rSrSrSS\S\S\S\S\S\S\S \S S 4U4S jjjrSS \S\ \ \S S 4Sjjr SSjr S \ R4SjrS\ RS \ R4SjrSS\S \4SjjrS \4U4SjjrSrU=r$)rrKrrrrrfan_in_fan_outr,returnNc >[[RU]5 [R"X40U D6 XlX lURX#XEXgX5 g)N)superrrr)rrd_active_adapterrU) r'rrKrGrrrrrrdr,r& __class__s r(r)Linear.__init__sJ bii')d9&9,+ ,Q[crr+ safe_merge adapter_namescLSSKJn U"X5nU(dgUGHnX@RR5;dM#UR 5nU(aUR R R5nURU5nXg- n[R"U5R5(d[SUS35eXeR lO0URU5nUR =R U- slURRU5 GM g)a Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`list[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. r)check_adapters_to_mergeNz1NaNs detected in the merged weights. The adapter z seems to be broken)peft.tuners.tuners_utilsrnr keysrr"datacloneget_delta_weightr<isfiniteallr6rrC)r'rkrlrnactive_adapterr orig_weights delta_weights r(merge Linear.merges E/D  +N!4!4!66!002 $.#4#4#9#9#?#?#AL#'#8#8#HL 0L >>,7;;==(OP^O__rs.:%%*#'#8#8#HL%%**l:*$$++N;),r+cUR(d[R"S5 g[UR5S:aURR 5nXR R5;a>URU5nUR5R=RU-sl [UR5S:aMgg)zG This method unmerges all merged adapter layers from the base weights. z Already unmerged. Nothing to do.Nr) mergedwarningswarnlenrpopr rprsrr"rq)r'rvrxs r(unmergeLinear.unmerges{{ MM< = $&&'!+!11557N!4!4!66#44^D ##%,,11\A1 $&&'!+r+c @URUnURUnURUnURUnURnUR nUR UnURUn URUn URS:H=(a- U[R:H=(d U[R:Hn U (a UR5nUR5nURn URn U nX-n[R "XS9nUX--nUX--n[R""XXU (a[R$OUS9nUR'SUR)S5R)S5R+SSU5UUUSS24R)S55 [R,"SUR/XX5R1S SSS 5R3XU5U5R3X5R4nU S :aGUR6nUR6nU (a UR5nUR5nUUU-- n[9UUR:5UR<U-nU (aUR?US 9nU$) an Compute the delta weight for GraLoRA adapter. GraLoRA applies block-wise low-rank adaptation with information exchange. This method computes the equivalent weight matrix that would be added to the base weight during merge. Args: adapter (str): The name of the adapter Returns: torch.Tensor: The delta weight matrix with shape [out_features, in_features] cpu)device)rdtyper3r4Nzikr, kro -> ikor)r) r r r r rrrrrr%r<float16bfloat16floatrr arangezerosfloat32scatter_ unsqueezeexpandeinsumviewpermutereshapeTr"rrdrto)r'adapterr r r r rrrrr cast_to_fp32rr gralora_ranksubblock_gralora_rank l_indices n_indices i_indicesgralora_A_scatteredrxweight_A_generalweight_B_generals r(rsLinear.get_delta_weightswNN7+ NN7+  227; 227;!!NN7+ ==) FF7O{{e+c%--1G1b5TYTbTbKb !)I!)I&& ((  , 9LL< +":; !9: #kk LXdu}}jo  $$    " , ,Q / 6 6r1l K iA- . 8 8 ;  LL!#((bAq!$>  W[ / Q  a<077 077 #3#9#9#; #3#9#9#;  ,/?? ?L!t/B/BCdllSZF[[  '???7Lr+xcURnUR(a9UR(aUR5 UR"U/UQ70UD6nGOUR(aUR"U/UQ70UD6nGOWUR"U/UQ70UD6nURnUR S:HnU(aUR S5nURGHnXRR5;aM#URUn URUn URUn URUn URUn U nURUnURUnUR UnUR"UnU RnUR$unnnUnUU-n[&R("S[&R("SU"UR+U55R-UUUUU-5U 5R-UUUUU5R/SSSSS5R1UUUUU-5U 5R1UUS5nU(aUR3S5nUUUR+U5-- nUS:dGMU "U "U"UR+U5555nU(aUR3S5nUUUR+U5-- nGM UR+U5nU$) Nr4r3zbljr, jro -> bljozblni, nir -> blnrrrr)rdisable_adaptersr|rrndimrrJr rpr r r rrrrrr#r<rrrrrsqueeze)r'rargsr&previous_dtyperesulttorch_result_dtypex_is_2drvr r r r rrrrdropoutr gralora_dtypeBLrNroutput hybrid_outputs r(forwardLinear.forwardAs  {{ __Q888F [[__Q888F__Q888F!' ffkGKKN"&"6"6!)<)<)>> NN>:  NN>: $($:$:>$J!$($:$:>$J!FF>*  NN>: ==8..~>,,~6 ) $%GG!1k(4(9%'LL+] 3499!Q;RSCST! T!Q1&;<WQ1a+WQ1a*?&?@ '!Q##^^A.F'FII.@$AAAa<$56GPQPTPTUbPcHd6e$fM(5(=(=a(@ g (8(89K(LLLF[#7^>* r+cg)NTra)r'rKs r(supports_lora_conversionLinear.supports_lora_conversionsr+c*>[TU]5nSU-$)Nzgralora.)rg__repr__)r'repris r(rLinear.__repr__sg Cr+)rhrd)rr3r.r4rFT)FN)reN)default)rWrXrYrZstrr^rr_r)rlistryrr<Tensorrsrrrr` __classcell__)ris@r(rrs !$$!ss  s  ssssss ss,(<(rsB  -3&~/>~/Bo RYY o r+