>:jRSSKJr SSKrSSKrSSKrSSKJr SSKJrJ r J r SSK r SSK J r SSKJ s Jr SSKJr SSKJrJr Sq\S5rSr"S S \5r"S S \ R45r"S S\5r"SS\ R4\5r"SS\ R4\5rg)) annotationsN)contextmanager)AnyOptionalUnion)Function)BaseTunerLayercheck_adapters_to_mergec+# 0nUR5HZup#UR5nU[R;a[RUX'[ U5[RU'M\ Sv UHOnUR5nX!;aX[RU'M/[RR US5 MQ g7f)a A context manager that will add each keyword argument passed to `os.environ` and remove them when exiting. Will convert the values in `kwargs` to strings and upper-case all the keys. Example: ```python >>> import os >>> from accelerate.utils import patch_environment >>> with patch_environment(FOO="bar"): ... print(os.environ["FOO"]) # prints "bar" >>> print(os.environ["FOO"]) # raises KeyError ``` N)itemsupperosenvironstrpop)kwargs existing_varskeyvalues S/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/boft/layer.pypatch_environmentr&s$Mlln iik "** !#CM e* 3 %  iik  +0BJJsO JJNN3 % sC C c[b[$SSKJn [RR [ 5n[SSS9 U"SUS3US3/SS 9nSSS5 Uq[$!,(df  Wq[$=f![aCn[R"S US 35 [R"S 5 SnSnAUq[$SnAff=f) Nr)loadgcc)CCCXXfbd_cudaz/fbd/fbd_cuda.cppz/fbd/fbd_cuda_kernel.cuT)namesourcesverbosez#Failed to load the CUDA extension: z, check if ninja is available.zHSetting boft_n_butterfly_factor to 1 to speed up the finetuning process.) _FBD_CUDAtorch.utils.cpp_extensionrrpathdirname__file__r Exceptionwarningswarn)rcurr_dirres r get_fbd_cudar+Js/wwx(H  %U 3$:%67H:E\9]^H4I 4 3I   ;A3>\]^ `aI  s5 A;A#A;# A8-A;8A;; C2CCc8\rSrSrSr\S5r\S5rSrg) FastBlockDiaggz Implements a custom autograd Function for a fast block diagonal operation using CUDA. This function is optimized for 4D tensors where the last two dimensions are equal, representing block diagonal matrices for efficient computation on CUDA devices. c`[5RU5SnURU5 U$)a The forward method for FastBlockDiag. Computes the block diagonal operation on the input tensor using a CUDA-optimized function. This method assumes that the input is a 4D tensor where the last two dimensions are equal, which represent the blocks to be diagonalized. Parameters: ctx: A context object that can be used to stash information for backward computation. input (Tensor): The input tensor of shape (N, D, H, H), where `N` is the batch size, `D` represents one additional dimension (In BOFT, the number of BOFT blocks), and `H` is the size of the square blocks along the last two dimensions (In BOFT, the block size). Returns: Tensor: The resulting tensor after applying the block diagonal operation, will have the shape (N, DxH, DxH). r)r+forwardsave_for_backward)ctxinputoutputs rr0FastBlockDiag.forwardos.&''.q1 e$ cZURun[5RX5SnU$)Nr) saved_tensorsr+backward)r2 grad_outputr3 grad_inputs rr9FastBlockDiag.backwards,$$!^,,[@C r6N) __name__ __module__ __qualname____firstlineno____doc__ staticmethodr0r9__static_attributes__r=r6rr-r-gs/,r6r-c6^\rSrSrSrSU4SjjrSrSrU=r$)MultiplicativeDropoutLayerz7 Implements the multiplicative dropout layer for BOFT. c.>[TU]5 Xlg)z Initializes the multiplicative dropout layer. Parameters: p (float): The probability of dropping out a block. Defaults to 0.0. N)super__init__p)selfrK __class__s rrJ#MultiplicativeDropoutLayer.__init__s r6cUR(GaXURSURS:wa [S5eURup#pE[R"SUS5R 5n[ URU-5nX7- n[R"[R"XqRS9[R"XRS9/5n U [R"U5RSUSS5n [R"X#SSURS9n XU'[R"XARS9RX#SS5n SU - U-X--nU$)a[ Applies multiplicative dropout to the input tensor. Parameters: x (Tensor): The input tensor of shape (N, D, H, H), where `N` is the batch size, `D` represents one additional dimension (In BOFT, the number of BOFT blocks), and `H` is the size of the square blocks along the last two dimensions (In BOFT, the block size). z4The last two dimensions of input should be the same!r)devicerR)trainingshape ValueErrortorchrandintitemintrKcatonesrTzerosrandpermvieweyerepeat) rLxNDH_n_randomnum_to_replace num_zerosmask full_mask eye_matrixs rr0"MultiplicativeDropoutLayer.forwards, ===wwr{aggbk) !WXXJA!}}Q40557H!!_N*I99ejjI5;;W`iqiqKrstDq)*//1a;D A!Qqxx@I"&h 1XX6==aAqIJY!#i&<r?r@rArBrJr0rD __classcell__rMs@rrFrFs##r6rFcv\rSrSrSrSrSrSSjrSrSSjr SSS jjr SSS jjr S r S r SS jrSrSrg) BOFTLayerz Implements the BOFT layer. )boft_Rboft_s)boft_block_sizeboft_block_num boft_dropoutc FXl0Ul0Ul[R"05Ul[R "05Ul[R "05UlSUl /Ul SUl X l UR5n[U[R5(aUR UR"pCON[U[R$5(aUR&UR(pCO[+S[-U535eX0lX@lg)z Initializes the BOFT layer. Note, currently only support linear layer and convolutional layer, with further support for other layers to be added soon. Parameters: base_layer: the pretrained model layer FTzUnsupported layer type N) base_layerrxrynn ModuleDictrz ParameterDictrvrw_disable_adaptersmerged_adapterscast_input_dtype_enabledrget_base_layer isinstanceLinear in_features out_featuresConv2d in_channels out_channelsrWtype)rLr|rrrs rrJBOFTLayer.__init__s%! MM"-&&r* &&r* !&!(,% ((* j")) , ,(2(>(> @W@W  BII . .(2(>(> @W@W6tJ7G6HIJ J&(r6cPXR;ag[R"S5 g)NGScaling operation for BOFT not supported! Automatically set scale to 1.)scalingr'r()rLadapterscales r set_scaleBOFTLayer.set_scales ,, &  _`r6cUS:XagURH8nX RR5;aM"[R"S5 M: g)NrRractive_adaptersrvkeysr'r(rLractive_adapters r scale_layerBOFTLayer.scale_layers? A: "22N[[%5%5%77 MMc d 3r6NcURH8nX RR5;aM"[R"S5 M: g)Nz?Unscaling operation for BOFT not supported! Keeping scale to 1.rrs r unscale_layerBOFTLayer.unscale_layers5"22N[[%5%5%77 MM[ \ 3r6c [5(d SUlSnOSUlUS- nUS:a[SUS-S35eUS:a [US9n O[R "5n UR R[R"X055 US:XaUS:waURU-S:wa[S URS US 35eUS:waYU[[R"U55:a[S US-S US 35eUSU--S:wa[SUSUS-S 35e[URU-5nOUS:waUS:XaURU-S:wa[S URSUS 35eUS:wasURUSU--:a"[SURSUS-SUS 35eURUSU---S:wa"[SURSUS-SUS 35e[URU-5nO [S5eUS:wa0US-S:wa[SUS35eUS-S:wa[SUS35e[R"US-URUR45n [US-5HQn UR!UR[USU -- 5[US- 5U5n UR#U 5n XU 'MS UR%SU SS9 [R&"[R("US-X2U55UR*U'[R&"[R,"[UR.5S55UR0U'UR3X5 X R4U'X0R6U'UR9U5 UR;UR<US9 g)zV Update the linear layer with trainable BOFT weights. Override for other layer types. FrRTr-You can only specify boft_n_butterfly_factor ! to be a positive integer number.rproz in_features (') must be divisible by boft_block_num ()!0Invalid combination of boft_n_butterfly_factor () and boft_block_num (boft_block_num (J) must be a multiple of 2 raised to the power of boft_n_butterfly_factor (() must be divisible by boft_block_size (z$Invalid combination of in_features (), boft_n_butterfly_factor () and boft_block_size (ZSomething went wrong, please report this error: https://github.com/huggingface/peft/issues) must be an even number!boft_block_size (boft_P persistentinference_modeN)r+fbd_cuda_availablerWrFr}Identityrzupdater~rr[mathlog2rXemptyrangeblock_butterfly_permperm2matregister_buffer Parameterr^rvr]rrwreset_boft_parametersrxry%_move_adapter_to_device_of_base_layer set_adapterr)rL adapter_namerxryboft_n_butterfly_factorrz init_weightsrrboft_dropout_layerPipermperm_mats r update_layerBOFTLayer.update_layers~~&+D #&' #&*D ##:A"= "Q &?@WZ[@[?\\}~  # !;l!K !#     /Q!RS a Na$7.0A5 #D$4$4#55\]k\llno'!+*S>1J-KK$JKbefKfJgg}M~NNPQ"Q(?%?@AE$*>*:;EF]`aFaEbbde"$"2"2n"DEO  !n&9/1Q6 #D$4$4#55]^m]nnpq'!+##!=T:T'UV$>t?O?O>PPlnEHInImJJabqarrtu##!=T:T'UVZ[[$>t?O?O>PPlnEHInImJJabqarrtu!!1!1_!DENl  #a '!Q& #3N3CC\!]^^"a' #4_4EE^!_`` KK014d6F6FHXHXY Z.23A,,  #na&A"BCZ[H[D\^uD}}T*HaD 4 XqU;$&LL KK/!3^Ve f%  L!%'LLC@Q@Q.=\*,:L) 22<@ --nMr6cUSLaY[RRURUSSS9 [RRURUSSS9 gXRR 5;amUSLaY[RR URU5 [RRURU5 g[SU<35eg) z Reset the BOFT parameters. Frp皙?)meanstdg?NTz$Unknown initialization init_weights=) r}initnormal_rvrwrzeros_ones_rW)rLrrs rrBOFTLayer.reset_boft_parametersus 5 GGOODKK 5CSO I GGOODKK 5CSO I  ;;++- -t#t{{<89 dkk,78 #H.sort_blocks5D!LLOM ;;q ;LLLD!,E<<4+DE=a8J#J/EEaAN LL  v0&2C%  PttDJJr6) rr|rvryrxrzrwrrrrrr)r| nn.ModulereturnNone)rfloatrrNrrFrbool)rR)r>r?r@rArBadapter_layer_namesother_param_namesrJrrrrrrrrrDr=r6rrtrts\ /M!)Fae] %lNlN\L"&%N r6rtc^\rSrSrSrS S U4SjjjrS S SjjrSSjrSSjrSSjr SU4Sjjr S r U=r $)riz$ BOFT implemented in a dense layer. c  >[T U]5 [R"X40U D6 XplX lUR X#XEXh5 Xlgr)rIrJrtfan_in_fan_out_active_adapterris_target_conv_1d_layer) rLr|rrxryrrzrrrrrMs rrJLinear.__init__sM 46v6,+  >T` (?$r6c[X5nU(dgUGHInX0RR5;dM#UR5nURR nU(Ga URR R5nURU5upx[R"USS5n[R"XvRUR 55n[R"USS5nXh-n[R"U5R5(d[SUS35eUR!5RU5UR"RlOURU5upxURR R5n[R"USS5n[R"XvRUR 55n[R"USS5nXh-nUR!5RU5UR"RlUR$R'U5 GML g) Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. NrrRz1NaNs detected in the merged weights. The adapter z seems to be broken)r rvrrweightrrcloneget_delta_weightrXrmmrisfiniteallrW contiguousr|rappend rL safe_merge adapter_namesrr| orig_dtype orig_weightbutterfly_oft_matrws rmerge Linear.merges0D  +N!1!1!33!002 '..44 #-"3"3"8"8">">"@K040E0En0U-%"'//+q!"DK"'((+e"fK"'//+q!"DK"-"6K >>+6::<<(OP^O__rs3>2H2H2J2M2Mj2YDOO**/040E0En0U-%","3"3"8"8">">"@K"'//+q!"DK"'((+e"fK"'//+q!"DK"-"6K2=2H2H2J2M2Mj2YDOO**/$$++N;=,r6cUR(d[R"S5 g[UR5S:GaEURR 5nUR 5nURRnXRR5;aURU5upEURRR5n[R"USS5n[R "UR#5UR%UR55n[R"USS5nUSU- -R%U5URl [UR5S:aGMDggzG This method unmerges all merged adapter layers from the base weights. z Already unmerged. Nothing to do.NrrR)mergedr'r(rrrrrrrvrrrrrXrrtrrLrr|r r"rwr!s runmergeLinear.unmerge!s*{{ MM< = $&&'!+!11557N,,.J#**00J!1!1!33,0,A,A.,Q)!(//44::< #ook1a@ #hh'8':':'ef #ook1a@ *5V*D)H)H)T !!&$&&'!++r6cURUnURUnURupEpgURXE-Xf5nUR U5nURXEXf5nUR (a[ RU5n OIURS5n[R"[R"U56n U RS5n URRU RU R 5n [R""XR%SSS55n [R""X5n U Sn ['SU RS5H n XU -n M X4$) Compute the delta weight for the given adapter. Args: adapter (str): The name of the adapter for which the delta weight should be computed. rrrR)rvrwrVr`rrr-applysqueezerX block_diagunbindrrrrTrbmmpermuterrLrrvrwrdrerfrgorth_rotate_butterflyblock_diagonal_butterflyrbutterfly_oft_mat_batchr"rs rrLinear.get_delta_weight7sNW%W%\\ aQUA) $ 1 1& 9 5 : :1 F  " "'4':':;P'Q $$9$A$A!$D !','7'7F[9\'] $'?'I'I!'L $ 8 ? ?AYA_A_`"')),DnnUVXY[\F]"^"'))F"L3A6q177:;A 7 :=N N <!((r6c URnUR(a9UR(aUR5 UR"U/UQ70UD6nGOUR(aUR"U/UQ70UD6nGOt[ R "URURUS9n[ R"[UR5S4URUS9nURGHnXRR5;aM#URUn URUn UR Un U R"uppU R%X-X5n UR'U 5nUR%XX5nU "U5nUR((a[*R-U5nOIUR/S5n[ R0"[ R2"U56nUR5S5nUR6R9U5nUR9U5n[ R:"UUR=SSS55n[ R:"UU5nUSn[?SUR"S5H nUUU-nM UU-nX-nGM UR9URA5RBRDR5nURA5RBRDn[ RF"USS5nUR9U5nUR9U5n[ RH"UU5n[ RF"USS5nUU-nUR9U5nURRJb4URRJR9U5URl%[LRN"UUURRJS9nUR9U5nU$)NrTrrRrr)r3rbias)(rdisable_adaptersr'r*r|rXrarrTr]r[rrrvrrwrzrVr`rrr-r.r/r0r1rrrr2r3rrrrrrr;Flinear)rLrcargsrprevious_dtyperesult boft_rotation boft_scalerrvrwdropoutrdrerfrgr5r6rr7r"rr!rotated_weightscaled_rotated_weights rr0Linear.forwardWsT  {{ __Q888F [[__Q888F!IId&6&6qxx~^MS):):%;Q$?XfgJ"&"6"6!)9)9);;^4^4++N;#\\ aQUA1(,(9(9&(A%(=(B(B1(N%(/0E(F%**/!q"9"?"?"BCA(?(BEV(V%D!2M A #0 =#7@T((*1166<<=A--/66;;K//+q!'>~'N$XXA.C$//J^J^_F>* r6c*>[TU]5nSU-$Nzboft.rI__repr__rLreprMs rrKLinear.__repr__g }r6)rrr)rrrFTF)rrrxr[ryr[rr[rzrrr rUnion[bool, str]rr rrFNrr rzOptional[list[str]]rrrrz!tuple[torch.Tensor, torch.Tensor]rc torch.Tensorr?rrrrrVrr) r>r?r@rArBrJr#r*rr0rKrDrqrrs@rrrs !'(!$)-(-?? ?  ? "% ???'?"&? ??00@r6rc^\rSrSrSrS S U4SjjjrS SSjjrSSSjjrSSjrSSjr SSjr SU4S jjr S r U=r $)riz% BOFT implemented in a Conv2d layer. c ~>[T U]5 [RX5 X lUR X#XEXg5 gr)rIrJrtrr) rLr|rrxryrrzrrrMs rrJConv2d.__init__s: 4,+  >T` r6c [5(d SUlSnOSUlUS- nUS:a[SUS-S35eUS:a [US9n O[R "5n UR R[R"X055 UR5n URU RS-U RS-n US:XaUS:waX-S:wa[S U S US 35eUS:waYU[[R"U55:a[S US-S US 35eUSU--S:wa[SUSUS-S 35e[X-5nOUS:wa}US:XawX-S:wa[S U SUS 35eUS:waIXSU--:a[SU SUS-SUS 35eXSU---S:wa[SU SUS-SUS 35e[X-5nO [S5eUS:wa0US-S:wa[SUS35eUS-S:wa[SUS35e[R "US-X45n [#US-5HGn UR%U [USU -- 5[US- 5U5nUR'U5nXU 'MI UR)SU SS9 [R*"[R,"US-X2U55UR.U'[R*"[R0"S[UR2555UR4U'UR7X5 X R8U'X0R:U'UR=U5 UR?UR@US9 g)z6 Update the conv2d layer with trainable BOFT weights. FrRTrrrrproz Convolutional kernel dimension (rrrrrrrrz7Invalid combination of convolutional kernel dimension (rrrrrrrrN)!r+rrWrFr}rrzrr~rr kernel_sizer[rrrXrrrrrrr^rvr]rrwrrxryrrr)rLrrxryrrzrrrrr|conv_filter_dimrrrrs rrConv2d.update_layers ~~&+D #&' #&*D ##:A"= "Q &?@WZ[@[?\\}~  # !;l!K !#     /Q!RS((* **Z-C-CA-FFI_I_`aIbb a Na$7/14 66GGno}n~AB'!+*S>1J-KK$JKbefKfJgg}M~NNPQ"Q(?%?@AE$*>*:;EF]`aFaEbbde"/"CDO  !n&90A5 66GGoppAACD'!+".=\*,:L) 22<@ --nMr6c|[X5nU(dgUGH"nX0RR5;dM#UR5nURR nU(Ga\URR R5nURU5upxURURURURS-URS-5n[R"USS5n[R"XvR!UR 55n[R"USS5nXh-nURURURURSURS5nUR#5R!U5UR$RlGOZURU5upxURR R5nURURURURS-URS-5n[R"USS5n[R"XvR!UR 55n[R"USS5nXh-nURURURURSURS5nUR#5R!U5UR$RlUR&R)U5 GM% g)rNrrR)r rvrrrrrrrr`rrr\rXrrrrr|rrrs rr# Conv2d.merge(s0D  +N!1!1!33!002 '..44 #-"3"3"8"8">">"@K040E0En0U-%"-"2"2))4+;+;j>T>TUV>W+WZdZpZpqrZs+s#K#(//+q!"DK"'((+e"fK"'//+q!"DK"-"6K"-"2"2))4+;+;Z=S=STU=VXbXnXnopXq#K3>2H2H2J2M2Mj2YDOO**/040E0En0U-%","3"3"8"8">">"@K"-"2"2))4+;+;j>T>TUV>W+WZdZpZpqrZs+s#K#(//+q!"DK"'((+e"fK"'//+q!"DK"-"6K"-"2"2))4+;+;Z=S=STU=VXbXnXnopXq#K3>2H2H2J2M2Mj2YDOO**/$$++N;O,r6c(UR(d[R"S5 g[UR5S:GaURR 5nUR 5nURRnXRR5;GaUURU5upEURRR5nURURUR UR"S-UR"S-5n[$R&"USS5n[$R("UR+5UR-UR55n[$R&"USS5nUSU- -nURURUR UR"SUR"S5nUR-U5URl [UR5S:aGMggr&)r'r'r(rrrrrrrvrrrrr`rrr\rXrrr(rr)s rr*Conv2d.unmergecs{{ MM< = $&&'!+!11557N,,.J#**00J!1!1!33,0,A,A.,Q)!(//44::< )..%%$$z'='=a'@@:CYCYZ[C\\ $ook1a@ #hh'8':':'ef #ook1a@ )QZ8 )..%%$$**1-**1-  *5 )C !!&/$&&'!++r6cURUnURURSS5nURupEpgUR XE-Xf5nUR U5nUR XEXf5nUR (a[RU5n OIURS5n[R"[R"U56n U RS5n URRU R U R"5n [R$"XR'SSS55n [R$"X5n U Sn [)SU RS5H n XU -n M X4$)r-rrRr)rvrwrrVr`rrr-r.r/rXr0r1rrrrTrr2r3rr4s rrConv2d.get_delta_weights[W%W%//15\\ aQUA) $ 1 1& 9 5 : :1 F  " "'4':':;P'Q $$9$A$A!$D !','7'7F[9\'] $'?'I'I!'L $ 8 ? ?AYA_A_`"')),DnnUVXY[\F]"^"'))F"L3A6q177:;A 7 :=N N <!((r6c  URnUR(a9UR(aUR5 UR"U/UQ70UD6nGOUR(aUR"U/UQ70UD6nGO^[ R "URURRS-URRS-URURS9n[ R"[UR5S4URURS9nURGHnXRR5;aM#URUn UR UR#SS5n UR$Un U R&uppU R)X-X5n UR+U 5nUR)XX5nU "U5nUR,(a[.R1U5nOIUR3S5n[ R4"[ R6"U56nUR9S5nUR:R=U5nUR=U5n[ R>"UURASSS55n[ R>"UU5nUSn[CSUR&S5H nUUU-nM UU-nX-nGM UR=URRDRFR5nURRDRFnUR)URURURRS-URRS-5n[ R""USS5n[ RH"UU5n[ R""USS5nUU-nUR)URURURRSURRS5nURKUUR5nURKURRLUR5n[NRP"UUUURRRSURRTSS9nUR=U5nU$)Nrr:rRr)r3rr;paddingstride)+rr<r'r*r|rXrarr\rTr]r[rrrvrrwrrzrVr`rrr-r.r/r0r1rrrr2r3rrrr_cast_input_dtyper;r=conv2drfrg)rLrcr?rr@rArBrCrrvrwrDrdrerfrgr5r6rr7r"rr!rErFr;s rr0Conv2d.forwards"  {{ __Q888F [[__Q888F!II  4??#>#>q#AADOOD_D_`aDbbxxggM S):):%;Q$?XYX_X_`J"&"6"6!)9)9);;^4^4>>q!D++N;#\\ aQUA1(,(9(9&(A%(=(B(B1(N%(/0E(F%**/!q"9"?"?"BCA(?(BEV(V%D!2M A #0 ;#7>T__++00667A//0055K%**!!  4??#>#>q#AADOOD_D_`aDbbK //+q!$>!!4#3#3T__5P5PQR5SUYUdUdUpUpqrUs% !&&q*?*E*EFA))$//*>*>@U@[@[\DXX,//2--a0 F>* r6c*>[TU]5nSU-$rIrJrLs rrKConv2d.__repr__rOr6)rr)rPrrrT)r|rrrrxr[ryr[rr[rzrrrQrrrrrRrSrrTrUrW)r>r?r@rArBrJrr#r*rr0rKrDrqrrs@rrrs !'(!)-       "%   '    6 %rNrNh9rvs$# %'' #L   & &F:#H#L22jM M `GRYY GTWRYY Wr6