>:j_SSKJr SSKrSSKJr SSKrSSKJr SSKJs J r SSK J r J r Jr SSKJr SSKJr "SS \ 5r"S S \R(\5r"S S \R*\5rg)) annotationsN)Optional)BaseTunerLayer_get_in_out_featurescheck_adapters_to_merge) transpose) BufferDictc\rSrSrSrSrSrSSjrSSjrSSjr SSSjjr SS jr SS jr SS jr SS jrSS jrSSjrSSjrSrg) TinyLoraLayeru TinyLoRA layer implementation. TinyLoRA is based on LoRA-XS and uses SVD decomposition of frozen weights. The key innovation is replacing the trainable r×r matrix R with: R = sum_i(v[i] * P[i]) where v is a tiny trainable vector and P_i are fixed random projection matrices. The forward pass computes: result += lora_B(R(lora_A(x))) where lora_A and lora_B are frozen SVD components. ) tinylora_v) tinylora_A tinylora_B tinylora_Pc ZXl0Ul0Ul[R"05UlSUl0Ul[0SS9Ul [0SS9Ul [0SS9Ul SUl /Ul SUl[UR!55uUlUlX lg)NT) persistentF) base_layerrunn ModuleDicttinylora_dropoutr_tinylora_v_refr rrr_disable_adaptersmerged_adapters _layer_idxrget_base_layer in_features out_featureskwargs)selfrr!s W/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/tinylora/layer.py__init__TinyLoraLayer.__init__1s$ " b 1489; %RD9$RD9%RD9"'!*..B4CVCVCX.Y+$+ c*[5nURURR55 URHFn[ XS5nUcM[ US5(dM'URUR55 MH [U5$)z4Return a sorted list of all available adapter names.Nkeys)setupdaterr(other_param_namesgetattrhasattrsorted)r" adapter_namesnameattrs r#_all_available_adapter_names*TinyLoraLayer._all_available_adapter_namesRsu T116689**D4t,DGD&$9$9$$TYY[1+m$$r&cXR;a URU URHn[XS5nUcMX;dMX1 M XR;a URU XR;a URU XR ;a UR U XR ;aUR SSnURU5 U(aURU5 gUR5nU(dUR/5 gUSn[R"SUSUS35 URU5 gg)z!Delete an adapter from the layer.NrzAdapter z< was active which is now deleted. Setting active adapter to .) rr+r,rrractive_adaptersremove set_adapterr2warningswarn)r" adapter_namer1 param_dictr6remaining_adaptersnew_active_adapters r#delete_adapterTinyLoraLayer.delete_adapter\s8 // /$$\2**D T2J%,*D,+ 66 !|$ 66 !|$ 00 0%%l3 // /"2215O  " "< 0  1%)%F%F%H")$$R();A)>&MM"<.0l-.a1$$%78 0r&cg)NT)r"r;s r#supports_lora_conversion&TinyLoraLayer.supports_lora_conversionsr&cXlg)zKSet the layer index, used for deterministic seeding of projection matrices.N)r)r"idxs r# set_layer_idxTinyLoraLayer.set_layer_idxsr&cZURbX R-$U[U5S--$)z>Get a deterministic seed for this layer's projection matrices.i')rhash)r"r; base_seeds r#_get_layer_seedTinyLoraLayer._get_layer_seeds. ?? &. .4 -555r&c URnURnURn URn URn US::a[ SU35eUS::a[ SU35eXpRU'US:a[ R"US9n O[ R"5n URR[ R"X055 X l X!UURU'URXU 5n XRU'URXX5 UR!U5 UR#UR$U S9 gz?Initialize layer with SVD decomposition and projection tensors.rz?`r` should be a positive integer value but the value passed is z?`u` should be a positive integer value but the value passed is )p)inference_modeN)rrprojection_seedrRfan_in_fan_out ValueErrorrDropoutIdentityr*rrr _init_svdr_init_projection%_move_adapter_to_device_of_base_layerr8r6)r"r;rv_keyrconfigr!rrrSrRrTtinylora_dropout_layeractual_rs r# update_layerTinyLoraLayer.update_layers8 HH!22 00.... 6^_`^abc c 6^_`^abc c | c !%'ZZ2B%C "%'[[] " $$R]]L3Y%Z[%-7-Ee-L\*>>,>B'| lxI 22<@ --nMr&cUR5nURRn[XS5nURnUR 5n[ RRUSS9upn [URSURS5n [X+5n USS2SU 24RU5n U SU RU5nU SU 2SS24RU5n[ R"U5nURS5U-R5URU'U URS5-R5UR U'U $)a Compute truncated SVD of base weights and store as frozen buffers. Compute truncated SVD and distribute singular values to both A and B: - W = U @ S @ V^T (full SVD) - We store: tinylora_A = diag(sqrt(S[:r])) @ V[:r, :] (shape: r x in_features) - We store: tinylora_B = U[:, :r] @ diag(sqrt(S[:r])) (shape: out_features x r) Distributing S equally avoids imbalanced norms between A and B. This allows: delta_W = tinylora_B @ R @ tinylora_A Returns: int: The actual rank used (may be less than r if matrix dimensions are smaller) F full_matricesrN)rweightdatardtypefloattorchlinalgsvdminshapetosqrt unsqueeze contiguousrr)r"r;rrTrrerg weight_fp32USVhmax_rankr^U_rS_rV_rsqrt_S_rs r#rXTinyLoraLayer._init_svds;((* ""''62 lln <<##Ku#Ebv||A Q8q#9H9 o  ' looe$(A!!%(::c? *2););A)>)D(P(P(R %),x/A/A!/D)D(P(P(R %r&cURX5n[R"5RU5n[R"SSUS-- X#U4US9nXpR U'g)u=Initialize fixed random projection tensors P ∈ R^{u×r×r}.rPg?g?)meanstdsize generatorN)rLri Generator manual_seednormalr)r"r;rrrKseedgenPs r#rYTinyLoraLayer._init_projectionsX##L<oo++D1 LLcsaf~Q1IQT U() %r&cURUnURUnURURURS9n[ R "SX#5nU$)z:Reconstruct R matrix from v and P: R = sum_i(v[i] * P[i]).devicergz i,ijk->jk)rrrnrrgrieinsum)r"r;vrRs r# _compute_RTinyLoraLayer._compute_RsS   . OOL ) DDD 0 LLa +r&c.URUnURUnURU5nURnURnUR XVS9nUR XVS9nUR S:H=(a- U[R:H=(d U[R:HnU(a0UR5nUR5nUR5nX4-U-n[USS5n [X5nU(aUR US9nU$)z Compute delta_W = tinylora_B @ R @ tinylora_A for merging. Returns weight update in the same shape as the base layer weight. For Conv1D layers (fan_in_fan_out=True), the result is transposed to match the (in_features, out_features) convention. rcpurTF)rg) rrrrrgrntyperifloat16bfloat16rhr,r) r"r;ABrrrg cast_to_fp32deltarTs r#get_delta_weightTinyLoraLayer.get_delta_weight s OOL ) OOL ) OOL ) DDD , DDD ,{{e+c%--1G1b5TYTbTbKb  A A A  !'7?%0 HH5H)E r&)rrrrrr!rr rrrrrrrN)r nn.Module)returnz list[str])r;strrNone)default)r;rrbool)rFint)r;rrKrrrr;rr nn.ModuleDictr[rrr)r;rrrrTrrr)r;rrrrrrKr)r;rr torch.Tensor)__name__ __module__ __qualname____firstlineno____doc__adapter_layer_namesr+r$r2r?rCrGrLr_rXrYrr__static_attributes__rBr&r#r r s{ *BB%&9P6-N-N"-N -N  -N^-^ * 'r&r cv^\rSrSrSrS U4SjjrS S SjjrS SjrS SjrSU4Sjjr Sr U=r $)Lineari3z&TinyLoRA implemented in a dense layer.c >[[RU]5 [R"X40UD6 UR UlX@lURUUUURU5 URSS5Ul g)Nis_target_conv_1d_layerF) superrrr$r rT_active_adapterr_rgetrr"rrr[r;r\r! __class__s r#r$Linear.__init__6st bii')t:6:$33+     HH   (.zz2KU'S$r&cF[X5nU(dgUGHnX0RR5;dM#UR5nU(aURR R 5nURU5nXV- n[R"U5R5(d[SUS35eXTRlO0URU5nUR=R U- slURRU5 GM g)a Merge the active adapter weights into the base weights. Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. N1NaNs detected in the merged weights. The adapter  seems to be brokenrrr(rrerfclonerriisfiniteallrUrappendr" safe_merger/active_adapterr orig_weights delta_weights r#merge Linear.mergeOs0D  +N!5!5!77!002 #-#4#4#9#9#?#?#AL#'#8#8#HL 0L >>,7;;==(OP^O__rs.:%%*#'#8#8#HL%%**l:*$$++N;%,r&cUR(d[R"S5 g[UR5S:aURR 5nXR R5;a>URU5nUR5R=RU-sl [UR5S:aMggz2Unmerge all merged adapters from the base weights.z Already unmerged. Nothing to do.Nr mergedr9r:lenrpoprr(rrrerfr"rrs r#unmergeLinear.unmergeu{{ MM< = $&&'!+!11557N!5!5!77#44^D ##%,,11\A1 $&&'!+r&cURnUR(a9UR(aUR5 UR"U/UQ70UD6nGOYUR(aUR"U/UQ70UD6nGO0UR"U/UQ70UD6nUR GH nX`R R5;aM#UR UnURUnURU5n URUn U "U5n U RUR5n U Rn URU 5nURU 5nU RU 5n [R"X5n [R"X5n [R"X5nX^-nGM URU5nU$N)rgdisable_adaptersrrrr6rr(rrrrnrFlinear)r"xargsr!previous_dtyperesultrrrrdropout x_droppedrhrs r#forwardLinear.forwardsh  {{ __Q888F [[__Q888F__Q888F"&"6"6!)=)=)??OON3OON3OON3//?#AJ %LL1 #))DDLDDLDDLHHY*HHQN9#7<>* r&c*>[TU]5nSU-$Nz tinylora.r__repr__r"reprs r#rLinear.__repr__g S  r&)rrTr rrrrr[rr;rrrFNrrr/zOptional[list[str]]rrrrrrrrrr) rrrrrr$rrrrr __classcell__rs@r#rr3s]0TT"T T  T T2$[TU]5 [R"X40UD6 X@lUR UUUUR U5 gr)rr$r rr_rrs r#r$Embedding.__init__sJ t:6:+     HH   r&c |URnURnURn URn US::a[ SU35eUS::a[ SU35eXpRU'US:a[ R "US9n O[ R"5n URR[ R"X055 X l X!UURU'URX5n XRU'URXX5 URU5 UR!UR"U S9 grO)rrrSrRrUrrVrWr*rrr_init_svd_embeddingrrYrZr8r6) r"r;rr[rr\r!rrrSrRr]r^s r#r_Embedding.update_layers- HH!22 00.. 6^_`^abc c 6^_`^abc c | c !%'ZZ2B%C "%'[[] " $$R]]L3Y%Z[%-7-Ee-L\*++L<'| lxI 22<@ --nMr&cUR5nURRnURnUR 5n[ R RUSS9upxn [URSURS5n [X*5n USS2SU 24RU5n USU RU5n U SU 2SS24RU5n[ R"U 5nURS5U-R5URU'XRS5-R5URU'U $)a Compute truncated SVD of embedding weights and store as frozen buffers. Embedding weight shape: (num_embeddings, embedding_dim) We treat this as W where: - W = U @ S @ V^T (full SVD) - tinylora_A = diag(sqrt(S[:r])) @ V[:r, :] (shape: r x embedding_dim) - tinylora_B = U[:, :r] @ diag(sqrt(S[:r])) (shape: num_embeddings x r) Returns: int: The actual rank used (may be less than r if dimensions are smaller) FrbrrdN)rrerfrgrhrirjrkrlrmrnrorprqrr)r"r;rrrergrrrsrtrurvr^rwrxryrzs r#rEmbedding._init_svd_embeddings-((* ""'' lln <<##Ku#Ebv||A Q8q#9H9 o  ' looe$(A!!%(::c?*2););A)>)D(P(P(R %),/A/A!/D)D(P(P(R %r&cF[X5nU(dgUGHnX0RR5;dM#UR5nU(aURR R 5nURU5nXV- n[R"U5R5(d[SUS35eXTRlO0URU5nUR=R U- slURRU5 GM g)z7Merge the active adapter weights into the base weights.Nrrrrs r#rEmbedding.merges/D  +N!5!5!77!002 #-#4#4#9#9#?#?#AL#'#8#8#HL 0L >>,7;;==(OP^O__rs.:%%*#'#8#8#HL%%**l:*$$++N;%,r&cUR(d[R"S5 g[UR5S:aURR 5nXR R5;a>URU5nUR5R=RU-sl [UR5S:aMggrrrs r#rEmbedding.unmerge6rr&cUR(a7UR(aUR5 UR"U/UQ70UD6$UR(aUR"U/UQ70UD6$UR"U/UQ70UD6nURHnXPR R 5;aM"UR UnURUnURU5nURUn URn URn URXS9nURXS9nURXS9n[R"X5n U "U 5n X-n X-nXN-nM U$)Nr)rrrrr6rr(rrrrrgrnr embedding)r"rrr!rrrrrrrrgafter_Bafter_Rrs r#rEmbedding.forwardBsI  {{ ??16t6v6 6 ;;??16t6v6 64T4V4"22N__%9%9%;;/A/A/A++N;G]]FLLEF0AF0AF0Akk!'Gg&GkGKE^F?3B r&c*>[TU]5nSU-$rrrs r#rEmbedding.__repr__prr&)rrrr)r;rrrrrrrrrr)rrrrrr$r_rrrrrrrrs@r#rrs5  "       *+N+N"+N +N  +NZ#J<4 B,\!!r&r) __future__rr9typingrritorch.nnrtorch.nn.functional functionalrpeft.tuners.tuners_utilsrrrpeft.utils.otherr _buffer_dictr r rModulerrBr&r#r s`# bb&%RNRj|!RYY |!~@! =@!r&