>:j'pSSKJr SSKrSSKJrJr SSKJrJr SSK J r SSK J r \"SS\ 55r g) ) annotationsN) dataclassfield)OptionalUnion) PeftConfig)PeftTypec^\rSrSr%Sr\"SSS0S9rS\S'\"S SS 0S9rS\S '\"S SS 0S9r S\S'\"SSS0S9r S\S'\"SSS0S9r S\S'\"SSS0S9r S\S'\"SSS0S9r S\S'\"S SS0S9rS\S'\"S SS!0S9rS\S"'\"S#SS$0S9rS%\S&'\"SSS'0S9rS(\S)'\"SSS*0S9rS+\S,'\"SSS-0S9rS.\S/'\"SSS00S9rS\S1'U4S2jrS3rU=r$)4TinyLoraConfiga This is the configuration class to store the configuration of a [`TinyLoraModel`]. TinyLoRA is an extremely parameter-efficient fine-tuning method based on the paper "Learning to Reason in 13 Parameters" (arXiv:2602.04118). It uses SVD decomposition of frozen weights and projects a tiny trainable vector through fixed random tensors. Paper: https://arxiv.org/abs/2602.04118 Args: r (`int`, *optional*, defaults to `2`): SVD rank for the frozen U, Sigma, V decomposition. The paper recommends r=2. u (`int`, *optional*, defaults to `64`): Trainable vector dimension per group. This controls the expressivity of the adaptation. Can be as low as 1-13 for extreme parameter efficiency. weight_tying (`float`, *optional*, defaults to `0.0`): Degree of weight tying across target modules, as a ratio between 0.0 and 1.0. Controls how many modules share the same trainable vector v. 0.0 means no sharing (each module has its own v). 1.0 means full sharing (all modules share one v). Values in between give partial sharing. projection_seed (`int`, *optional*, defaults to `42`): Random seed for generating the fixed projection matrices P. save_projection (`bool`, *optional*, defaults to `True`): Whether to save the projection tensors P in the state dict. If False, they will be regenerated from the seed when loading. init_v_bound (`float`, *optional*, defaults to `0.02`): Uniform initialization bound for the trainable vector v. Values are initialized in [-init_v_bound, init_v_bound]. target_modules (`Union[List[str], str]`, *optional*): The names of the modules to apply TinyLoRA to. This can be a list of module names (e.g. `['q_proj', 'v_proj']`), a regex pattern (e.g. `'.*decoder.*(q|v)_proj$'`), or the special keyword `"all-linear"` to target all linear modules. Only `nn.Linear`, `nn.Embedding`, and `transformers.pytorch_utils.Conv1D` layers are supported. tinylora_dropout (`float`, *optional*, defaults to `0.0`): The dropout probability for TinyLoRA layers. fan_in_fan_out (`bool`, *optional*, defaults to `False`): Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses `Conv1D` which stores weights like (fan_in, fan_out). bias (`str`, *optional*, defaults to `"none"`): Bias type for TinyLoRA. Can be 'none', 'all' or 'tinylora_only'. modules_to_save (`List[str]`, *optional*): List of modules apart from TinyLoRA layers to be set as trainable and saved. init_weights (`bool` | `Literal["uniform"]`, *optional*, defaults to `True`): How to initialize the trainable vector v. Passing `True` (default) initializes v to zeros, making the adapter a no-op (identity operation). Passing `"uniform"` initializes v with uniform random values in `[-init_v_bound, init_v_bound]`. Passing `False` leaves v uninitialized (for advanced use cases). layers_to_transform (`Union[List[int], int]`, *optional*): The layer indexes to transform. If specified, only these layers will be adapted. layers_pattern (`Optional[Union[List[str], str]]`, *optional*): The layer pattern name, used only if `layers_to_transform` is different from `None`. Example: ```python from peft import get_peft_model, TinyLoraConfig config = TinyLoraConfig( r=2, # SVD rank (paper recommends 2) u=64, # Trainable vector dimension weight_tying=0.0, # No weight tying (0.0 = none, 1.0 = full) target_modules=["q_proj", "v_proj"], projection_seed=42, ) model = get_peft_model(base_model, config) ``` helpzTinyLoRA SVD rank (frozen))defaultmetadataintr@z$Trainable vector dimension per groupuzzDegree of weight tying across target modules (0.0 to 1.0). 0.0 = no sharing, 1.0 = full sharing (all modules share one v).float weight_tying*zRandom seed for generating the fixed projection matrices P. Used for initialising projections for new models or when loading a checkpoint that did not include these projections.projection_seedTzWhether to save the projection tensors P in the state dict. If False, they will be regenerated from the seed when loading. Setting to True increases checkpoint size but guarantees reproducibility across system configurations.boolsave_projectiong{Gz?z+Uniform init bound for v in [-bound, bound] init_v_boundNaList of module names, regex expression, or the keyword 'all-linear' to replace with TinyLoRA. For example, ['q_proj', 'v_proj'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'. Only nn.Linear, nn.Embedding, and transformers.pytorch_utils.Conv1D layers are supported.zOptional[Union[list[str], str]]target_moduleszTinyLoRA dropouttinylora_dropoutFzMSet this to True if the layer to replace stores weight like (fan_in, fan_out)fan_in_fan_outnonez?Bias type for TinyLoRA. Can be 'none', 'all' or 'tinylora_only'strbiasaList of modules apart from TinyLoRA layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.zOptional[list[str]]modules_to_savezHow to initialize the trainable vector v. True (default) initializes v to zeros, making the adapter a no-op. 'uniform' initializes v with uniform random values. False leaves v uninitialized.zUnion[bool, str] init_weightszThe layer indexes to transform. If this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index.zOptional[Union[list[int], int]]layers_to_transformzThe layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.layers_patternc>[TU]5 [RUl[ UR [5(a[UR 5O UR UlUR(aUR(d [S5eUR(d[R"S5 URS::a[SUR35eUR S::a[SUR 35eSUR"s=::aS::dO [SUR"35eg) NzQWhen `layers_pattern` is specified, `layers_to_transform` must also be specified.aSpecified to not save projection tensors P within the state dictionary. They will be restored using the PRNG key stored in `config.projection_seed`. Consider setting `config.save_projection` to `True` to guarantee restoring the checkpoint correctly on all system configurations.rz?`r` should be a positive integer value but the value passed is z?`u` should be a positive integer value but the value passed is rg?zM`weight_tying` should be a float between 0.0 and 1.0 but the value passed is )super __post_init__r TINYLORA peft_type isinstancerlistsetr&r% ValueErrorrwarningswarnrrr)self __class__s X/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/tinylora/config.pyr)TinyLoraConfig.__post_init__s  !**(243F3F(M(MC## $SWSfSf    t'?'?pq q## MMj  66Q;^_c_e_e^fgh h 66Q;^_c_e_e^fgh ht((/C/_`d`q`q_rs 0)r+r)__name__ __module__ __qualname____firstlineno____doc__rr__annotations__rrrrrrrrr"r#r$r%r&r)__static_attributes__ __classcell__)r3s@r4r r s?B10L'M NAsN 21W(X YAsY R L%! r OS" K  OT  @m7noL%o6; l  7N3 $C6CU:VWeW ijND&*k!lD#,1 j  ,O( &+ u &L"rEs7#("" qZq qr6