>:jCSSKJr SSKrSSKrSSKJrJrJr SSKrSSK J r SSK J s J r SSKJrJr "SS\5r"SS\ R$\5rg) ) annotationsN)AnyOptionalUnion)BaseTunerLayercheck_adapters_to_mergec\rSrSrSrSrSSjrSSSjjrSSjrSSjr SSjr SS jr SS jr SSS jjr S rg ) MissLayer) miss_block)miss_r miss_dropout miss_mini_rc Xl0Ul[R"05Ul0Ul[R "05UlSUl/Ul SUl X l UR5n[U[R5(a$URUR sUlUlg[#S[%U535e)NFTzUnsupported layer type ) base_layerr nn ModuleDictrr ParameterDictr _disable_adaptersmerged_adapterscast_input_dtype_enabledkwargsget_base_layer isinstanceLinear in_features out_features ValueErrortype)selfrrs S/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/miss/layer.py__init__MissLayer.__init__!s$ MM"-**2.!&!(,% ((* j")) , ,2<2H2H*JaJa /D d/6tJ7G6HIJ Jc US::a[SU35eX RU'X0RU'US:a[R"US9nO[R "5nXR U'UR5n [U [R5(aA[R"[R"X R5SS9URU'O![S[!U 5R"35eUS:XaCUR$U-S:wdURU-S:wa [S 5eUR'X5 OaUS :Xa1URU-S:wa [S 5eUR)XU5 O*U(aUR+X5 OUR-U5 UR/U5 UR1UR2US 9 g )zInternal function to create miss adapter Args: adapter_name (`str`): Name for the adapter to add. r (`int`): Rank for the added adapter. init_weights (`bool`): Whether to initialize weights. rz?`r` should be a positive integer value but the value passed is )pT requires_gradz0MiSS is not implemented for base layers of type batz=The weight matrix must be fully divisible into [r, r] blocks.minizmini_r is divided along the out_features dimension. For optimal performance and implementation simplicity,it is recommended that out_features be divisible by mini_r.Error: {self.out_features} % mini_r != 0)inference_modeN)rr rrDropoutIdentityrrrr Parametertorchzerosrr TypeErrorr__name__rreset_bat_parametersreset_mini_parametersreset_miss_parametersreset_miss_parameters_random%_move_adapter_to_device_of_base_layer set_adapteractive_adapters) r adapter_namermini_rr init_weightsr,rmiss_dropout_layerrs r! update_layerMissLayer.update_layer4s" 6^_`^abc c$% L!)/& # !#l!; !# *<,'((* j")) , ,,.LLQHYHY9Zjn,oDOOL )NtT^O_OhOhNijk k 5 !#q(D,=,=,AQ,F !`aa  % %l 6 V #  6)Q. ?  & &| ?   & &| 7  - -l ; 22<@ --nMr$c[R"[R"X R5SS9UR U'gNTr(rr/r0r1rr r r;r<s r!r6MissLayer.reset_miss_parametersms+(* U[[DUDU5Vfj(k %r$c[R"[R"URU-X"5SS9UR U'grCrDrEs r!r4MissLayer.reset_bat_parametersps4(* U[[ARARVWAWYZ5^nr(s %r$cp[R"[R"X#5SS9URU'grC)rr/r0r1r )r r;r<r=s r!r5MissLayer.reset_mini_parametersss%(* U[[5K[_(` %r$c[RRURU[R "S5S9 g)N)a)rinitkaiming_uniform_r mathsqrtr r;s r!r7&MissLayer.reset_miss_parameters_randomvs*   !>$))A, Or$cUS:XagURH8nX RR5;aM"[R"S5 M: g)NzGScaling operation for MiSS not supported! Automatically set scale to 1.r:r keyswarningswarnr scaleactive_adapters r! scale_layerMissLayer.scale_layerys? A: "22N__%9%9%;; MMc d 3r$NcURH8nX RR5;aM"[R"S5 M: g)Nz?Unscaling operation for MiSS not supported! Keeping scale at 1.rVrZs r! unscale_layerMissLayer.unscale_layers5"22N__%9%9%;; MM[ \ 3r$) rrrrrrr rrr r)rz nn.ModulereturnNoneF) r;strr<intr=rfr>z bool | strr,boolrbrc)r;re)r[floatrbrcNrbrc)r3 __module__ __qualname____firstlineno__adapter_layer_namesother_param_namesr"r@r6r4r5r7r]r`__static_attributes__r$r!r r s)AK4 %7N7N 7N 7N ! 7N7N 7NrltaPe]]r$r c^\rSrSrSrS S U4SjjjrSSSjjrSSjrSSSjjrSSSjjr SSjr SSS jjr SU4S jjr S r U=r$) MissLinearz$ MiSS implemented in a dense layer. c >[TU]5 [R"X40UD6 X lUR"X#XEU40UD6 X`lgri)superr"r _active_adapterr@miss_fn) r rr;r<r=rr>r __class__s r!r"MissLinear.__init__sF 46v6+ ,6XQWX# r$cx[X5nU(dgUGHnX0RR5;dM#UR5nURR nU(GaURR R5nURS:XaURX65nXg- nOsURS:Xa2URX0RRR 5nUnO1URX0RRR 5nUn[R"U5R5(d[SUS35eUR!U5URlGOURS:Xa^URX0RRR 5nUR=R UR!U5- slOURS:XaPURX0RRR 5nUR!U5URlOOURX0RRR 5nUR!U5URlUR"R%U5 GM g)a Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If `None`, all active adapters will be merged. Defaults to `None`. Nr*r+z1NaNs detected in the merged weights. The adapter z seems to be broken)rr rWrweightdtypedataclonerxget_delta_weightget_delta_weight_missrr0isfiniteallrtorappend)r safe_merge adapter_namesr\r orig_dtype orig_weight delta_weights r!mergeMissLinear.merges0D  +N!5!5!77!002 '..44 #-"3"3"8"8">">"@K||u,'+'<'<^'Y #3 /'+'A'A.RaRaRhRhRmRm'n &2 '+'A'A.RaRaRhRhRmRm'n &2  >>+6::<<(OP^O__rs.9^^J-GJ%%*||u,'+'<'<^__McMcMhMh'i "))..,//*2MM./'+'A'A.RaRaRhRhRmRm'n 1=1L )).'+'A'A.RaRaRhRhRmRm'n 1=1L )).$$++N;E,r$cUR(d[R"S5 g[UR5S:GaURR 5nUR 5nURRnXRR5;aUR 5RRR5nURS:XaURXSS9nO1URS:XaURXSS9nOURXSS9nUR!U5URl [UR5S:aGMgg)zG This method unmerges all merged adapter layers from the base weights. z Already unmerged. Nothing to do.Nrr*T)rer+)mergedrXrYlenrpoprr|r}r rWr~rrxrrr)r r\rrrrs r!unmergeMissLinear.unmerges {{ MM< = $&&'!+!11557N,,.J#**00J!5!5!77"113::??EEG <<5(#'#8#8Y]#8#^L\\V+#'#=#=n^b#=#cL#'#=#=n^b#=#cL)5)D !!&$&&'!++r$cURURnURURnURS:H=(a- U[R :H=(d U[R :HnURUnU(aUR5nURUR5nURS5nU(aURURS5U-XRS5U-U5RSSSS5n [R"URS55RUR5n [R"X-5n U RUR5n X- U -n U RSSSS5R"UR6n OzURURS5U-XRS5U-U5RSSSS5U-U-n U RSSSS5R"UR6n U(a2U RUS9n URU5URUlU $) Compute the delta weight for the given adapter. Args: adapter (str): The name of the adapter for which the delta weight should be computed. cpurrUr})r devicer}rr0float16bfloat16rhrsizereshapepermuteeyeinverseshaper~)r adapterrrrr} cast_to_fp32 weight_missr<oone inv_I_plus_bw output_tensors r!rMissLinear.get_delta_weights-)00(..{{e+c%--1G1b5TYTbTbKb oog. %++-K!nn[%6%67   R  ##K$4$4Q$71$:D||v-#{{1doo.J.JdN^N^_mNn.no"//?G ! AvvbzA~*()FF2JN(:a'? EE!a%67..q**=A#eii 0B0B0eAGGCRL0eRSRXRXY[R\`aRa0ecd0ekm&nqu&uuF';>* r$c URS:H$)Nr*)rxrRs r!supports_lora_conversion#MissLinear.supports_lora_conversions||u$$r$c*>[TU]5nSU-$)Nzmiss.)rv__repr__)r reprys r!rMissLinear.__repr__sg }r$)rwrx)rrr&T) r;rer<rfr=rfrrhr>zUnion[bool, str]rbrc)FN)rrgrzOptional[list[str]]rbrcrjrd)rrgrb torch.Tensor)rrrrrrrbr)default)r;rerbrg)rbre)r3rkrlrm__doc__r"rrrrrrrrp __classcell__)rys@r!rsrss!)-$$  $  $  $'$ $$ 4rsJ# '' Lm]m]`BIBr$