>:j,:hSSKJr SSKJrJr SSKJrJr SSKJ r SSK J r \"SS\ 55r g) ) annotations) dataclassfield)OptionalUnion) PeftConfig)PeftTypecF^\rSrSr%Sr\"SSS0S9rS\S'\"S SS 0S9rS\S '\"S SS 0S9r S\S'\"SSS0S9r S\S'\"SSS0S9r S\S'\"SSS0S9r S\S'\"SSS0S9r S\S'\"SSS0S9rS\S'\"SSS0S9rS\S '\"S!SS"0S9rS#\S$'U4S%jrS&rU=r$)' LilyConfigu This is the configuration class to store the configuration of a [`LilyModel`]. Args: r (`int`): Lily's rank. Determines the inner hidden dimension of each adapter and the rank of the weight update `A @ B`. In Lily, since the number of adapters is typically smaller than in LoRA, each adapter needs to carry more capacity, so it is recommended to use a larger `r` than in LoRA — typically `2x`, `3x`, or `4x` the LoRA rank you would normally use. The total number of trainable parameters scales with `r * (total_layers / stride_A + num_B)`, so increasing `r` while keeping `stride_A` large and `num_B` small is the recommended trade-off. stride_A (`int`): The number of consecutive layers that share one A adapter. For example, if `stride_A=4`, every 4 adjacent layers share the same A adapter, resulting in `total_layers / stride_A` distinct A adapters in total. The A adapter compresses the input into a low-rank representation of size `r`. `stride_A` should be no less than 1. Suggested values: `2`, `3`, or `4` (i.e. sharing every 2, 3, or 4 layers). Keeping `stride_A` large (fewer distinct A adapters) and increasing `r` instead leads to better performance than the opposite trade-off (small `stride_A`, small `r`). Setting `stride_A=1` means every layer has its own A adapter. NOTE: the A sharing happens within each target (layers with the same target suffix). For example, if your target_modules are `['q_proj', 'v_proj']` and you set `stride_A=2`, then every 2 adjacent q_proj layers will share an A adapter, and every 2 adjacent v_proj layers will share another A adapter, but the q_proj and v_proj layers will not share A adapters with each other since they have different suffixes. num_B (`int`): The number of shared B adapters. Unlike A adapters (which are grouped by layer), all B adapters are shared globally across every layer. For each forward pass, a router computes a weighted combination of all `num_B` B adapters (using softmax-normalized weights) to produce a single combined B adapter, which then projects the low-rank representation back to the original dimension. It is recommended to set `num_B` to a similar order as `total_layers / stride_A`. Suggested values: `total_layers / 2`, `total_layers / 3`, or `total_layers / 4`. Similar to `stride_A`, prefer smaller `num_B` with larger `r` over larger `num_B` with smaller `r`. NOTE: to train the router, you need at least 2 B adapters (i.e. `num_B >= 2`), since the router learns to compute a weighted combination of the B adapters. NOTE: the B sharing happens within each target (layers with the same target suffix). For example, if your target_modules are `['q_proj', 'v_proj']` and you set `num_B=4`, then there will be 4 B adapters shared across all q_proj layers, and another 4 B adapters shared across all v_proj layers, but the q_proj and v_proj layers will not share B adapters with each other since they have different suffixes. target_modules (`Union[List[str], str]`, *optional*): The names of the modules to apply Lily to. Can be a list of module name strings (e.g. `['q_proj', 'v_proj']`) or a regex pattern (e.g. `'.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'`). If not specified, Lily will be applied to all supported linear layers. scaling (`float`): A scalar multiplier applied to the combined adapter output (`scaling * A @ combined_B`) before adding it to the frozen weight's forward pass. Unlike LoRA, Lily does not use an `alpha / r` formulation; instead, `scaling` is a direct multiplier. This design makes it straightforward to sweep over values on a log scale (e.g. `0.01`, `0.1`, `1.0`, `10.0`). The optimal value is task-dependent and should be treated as a hyperparameter. We recommend starting with `1.0`. modules_to_save (`List[str]`, *optional*): List of modules apart from Lily layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved. exclude_modules (`Union[List[str], str]`, *optional*): The names of the modules to not apply the adapter. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. layers_to_transform (`Union[list[int], int]`, *optional*): The layer indexes to transform, if this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index. layers_pattern (`Optional[Union[List[str], str]]`, *optional*): The layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`. init_weights (`bool`): Whether to initialize Lily adapter weights using the default initialization scheme: A matrices are initialized with Kaiming uniform, and B matrices are initialized to zero, ensuring that the adapter output is zero at the start of training and does not disturb the pretrained model. It is strongly recommended to keep this as `True` unless you have a specific reason to change it. helpuLily's rank. Determines the inner hidden dimension of each adapter and the rank of the weight update `A @ B`. In Lily, since the number of adapters is typically smaller than in LoRA, each adapter needs to carry more capacity, so it is recommended to use a larger `r` than in LoRA — typically `2x`, `3x`, or `4x` the LoRA rank you would normally use. The total number of trainable parameters scales with `r * (total_layers / stride_A + num_B)`, so increasing `r` while keeping `stride_A` large and `num_B` small is the recommended trade-off.)defaultmetadataintraThe number of consecutive layers that share one A adapter. For example, if `stride_A=4`, every 4 adjacent layers share the same A adapter, resulting in `total_layers / stride_A` distinct A adapters in total.The A adapter compresses the input into a low-rank representation of size `r`. `stride_A` should be no less than 1.Suggested values: `2`, `3`, or `4` (i.e. sharing every 2, 3, or 4 layers). Keeping `stride_A` large (fewer distinct A adapters) and increasing `r` instead leads to better performance than the opposite trade-off (small `stride_A`, small `r`). Setting `stride_A=1` means every layer has its own A adapter.Note: A sharing happens within each target module type independently. For example, if `target_modules=['q_proj', 'v_proj']` and `stride_A=2`, then every 2 adjacent `q_proj` layers share one A adapter and every 2 adjacent `v_proj` layers share another A adapter, but `q_proj` and `v_proj` layers never share A adapters with each other.stride_AapThe number of shared B adapters. Unlike A adapters (which are grouped by layer), all B adapters are shared globally across every layer. For each forward pass, a router computes a weighted combination of all `num_B` B adapters (using softmax-normalized weights) to produce a single combined B adapter, which then projects the low-rank representation back to the original dimension. It is recommended to set `num_B` to a similar order as `total_layers / stride_A`. Suggested values: `total_layers / 2`, `total_layers / 3`, or `total_layers / 4`. Similar to `stride_A`, prefer smaller `num_B` with larger `r` over larger `num_B` with smaller `r`. NOTE: to train the router, you need at least 2 B adapters (i.e. `num_B >= 2`), since the router learns to compute a weighted combination of the B adapters.Note: B sharing happens within each target module type independently. For example, if `target_modules=['q_proj', 'v_proj']` and `num_B=4`, then there will be 4 B adapters shared across all `q_proj` layers and another 4 B adapters shared across all `v_proj` layers, but `q_proj` and `v_proj` layers never share B adapters with each other.num_Bg?aA scalar multiplier applied to the combined adapter output (`scaling * A @ combined_B`) before adding it to the frozen weight's forward pass. Unlike LoRA, Lily does not use an `alpha / r` formulation; instead, `scaling` is a direct multiplier. This design makes it straightforward to sweep over values on a log scale (e.g. `0.01`, `0.1`, `1.0`, `10.0`). The optimal value is task-dependent and should be treated as a hyperparameter. We recommend starting with `1.0`.floatscalingNzList of module names or regex expression of the module names to replace with Lily. For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'. If not specified, Lily will be applied to all supported linear layers.zOptional[Union[list[str], str]]target_modulesa"List of module names or regex expression of the module names to exclude from Lily. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings.exclude_modulesaList of modules apart from Lily layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved.zOptional[list[str]]modules_to_savezThe layer indexes to transform, if this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index.zOptional[Union[list[int], int]]layers_to_transformzThe layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is often called `'layers'` or `'h'`.layers_patternTa{Whether to initialize Lily adapter weights using the default initialization scheme: A matrices are initialized with Kaiming uniform, and B matrices are initialized to zero, ensuring that the adapter output is zero at the start of training and does not disturb the pretrained model. It is strongly recommended to keep this as `True` unless you have a specific reason to change it.bool init_weightsc&>[TU]5 [RUl[ UR [5(a[UR 5O UR Ul[ UR[5(a[UR5O URUl UR(aUR(d [S5eURS:a [S5eURS:a [S5eg)NzQWhen `layers_pattern` is specified, `layers_to_transform` must also be specified.rz`stride_A` must be at least 1.rz8`num_B` must be at least 2 for the router to be trained.)super __post_init__r LILY peft_type isinstancerlistsetrrr ValueErrorrr)self __class__s T/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/lily/config.pyr"LilyConfig.__post_init__s !(243F3F(M(MC## $SWSfSf *4D4H4H$)O)OC$$ %UYUiUi    t'?'?pq q ==1 => > ::>WX X )rr$r)__name__ __module__ __qualname____firstlineno____doc__rr__annotations__rrrrrrrrrr"__static_attributes__ __classcell__)r*s@r+r r sBH s   As   [ Hc(  c E3* D   GU 7< Y  7N3 8= 2  8O4 ,1 z  ,O( r<s7#("" UYUY UYr-